Collaborative storage architecture

By realizing direct communication and data transmission between solid-state drivers in the computing system, and using the flash conversion layer to operate in a collaborative manner, the problems of data transmission complexity and load increase in the computing system are solved, and more efficient storage management and horizontal scalability are achieved. sex.

CN113515236BActive Publication Date: 2024-11-12GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110790098.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-03-22
Filing Date
2021-07-13
Publication Date
2024-11-12
Estimated Expiration
2041-07-13

AI Technical Summary

Technical Problem

In computing systems, as the size of the storage device increases, the computing load and memory bandwidth load of the host computing device significantly increases, limiting the further growth of the number of solid-state drivers and storage device capacity, resulting in complex and congested communication and data transmission.

Method used

By realizing direct communication and data transmission between data storage drives in a computing system, using the Flash Transformation Layer (FTL) to operate in a coordinated manner between various solid-state drives, reducing dependence on host computing devices, and using an interconnect link protocol to allow data Device-to-device (D2D) communication between storage drives reduces the computing load and memory bandwidth load of the host computing device.

Benefits of technology

It realizes more efficient data transmission and storage management, supports greater storage capacity growth, reduces the load on host computing devices, and improves the lateral scalability and data transmission efficiency of computing systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113515236B_ABST
    Figure CN113515236B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a collaborative storage architecture. An interconnect architecture enables communication and / or data transfer between data storage drives in a computing system. A flash translation layer (FTL) in each data storage drive can operate in a collaborative manner, which allows communication and / or data transfer across the memory array of each of the data storage drives implemented in the computing system. Direct communication and / or data transfer between data storage drives in the computing system can be enabled without deferring back to a host computing device in the computing system. Therefore, the computing load on the host computing device is reduced and the flexibility of expanding storage devices in the computing system is increased.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a collaborative storage architecture. Background Art

[0002] Compared to hard disk drives (HDDs), solid-state drives (SSDs) based on NAND flash memory are widely used in various applications due to their higher bandwidth and lower latency. Generally, SSDs provide faster access times and lower power consumption than comparable HDDs. Interconnect protocols are often used between host computing devices and solid-state drives (SSDs) to facilitate signal and data transmission. The host computing device acts as a server that provides information resources, services, commands, and / or applications to users or other nodes on the network. The host computing device manages the SSD to access data from the SSD, such as reading or writing data for operation and storage, and provides services such as automatic tiering and global wear level management.

[0003] In recent years, multiple solid-state drives (SSDs) are often used to provide multiple storage arrays to build large data storage devices in computing systems. As a result, the communication and / or connection between a central host (such as a host computing system) and a solid-state drive (SSD) becomes complicated and sometimes congested. Since data communication and / or signal transmission from a solid-state drive (SSD) needs to be delayed back to the host computing system for management under the current interconnection protocol, the computing load and memory bandwidth load added to the host computer device increase significantly as the size of the storage device grows. When the host computing device reaches its maximum computing load and memory bandwidth load, the number of solid-state drives allowed under the host computing device also reaches its limit, thereby limiting the further growth of the capacity of the storage device. Summary of the invention

[0004] The present disclosure provides an interconnect architecture that enables communication and / or data transfer between data storage drives in a computing system with minimal involvement from a host computing device in the computing system. A flash translation layer (FTL) in each data storage drive can operate in a collaborative manner that allows communication and / or data transfer across memory arrays from each of the data storage drives implemented in the computing system. In one example, an aspect of the present disclosure described herein includes a method for processing commands in a computing system. The method includes: receiving an operation command in a first data storage drive disposed in a computing system by one or more processors; in response to the operation command, the first data storage drive directly communicates with a second data storage drive disposed in the computing system; and determining, based on the communication, by one or more processors, a data storage drive in the computing system to process the operation command.

[0005] In one example, the determination of the data storage drive is further based on the type of the operation command. When the operation command is a read command, the determination of the data storage drive is based on the flash translation layer indicating where the data resides in the first data storage drive and / or the second data storage drive. When the operation command is a write command, the determination of the data storage drive is based on the allocation table including at least one of a global drive attribute or a global drive wear statistic. The global drive attribute includes at least one of a capacity or a wear capability. The direct communication with the second data storage drive further includes direct communication without communicating with a host computing device in the computing system. The first and second data storage drives are solid-state drives. The operation command includes a data write command, a data read command, a data programming command, or a data erase command.

[0006] In one example, receiving an operation command in a first data storage drive further includes: processing the operation command by one or more processors through a first buffer manager configured in the first data storage drive; and directly communicating by the one or more processors from the first buffer manager to a second buffer manager configured in the second data storage drive.

[0007] In one example, the data block is defined in a plurality of non-volatile memories or a plurality of volatile memories arranged in a first data storage drive or a second data storage drive. The plurality of non-volatile memories are a plurality of NAND flash memories. In another example, the plurality of volatile memories are a plurality of DRAMs. The direct communication between the first solid-state drive and the second solid-state drive is transmitted via an interconnect link protocol defined between the first solid-state drive and the second solid-state drive. The interconnect link protocol enables direct communication between the first data storage drive and the second data storage drive without communicating with a host computing device in a computing system.

[0008] Another aspect of the present disclosure includes an apparatus. The apparatus includes a computing system, the computing system including a host computing device. A plurality of data storage drives communicate with the host computing device. An interconnect link protocol is connected to the plurality of data storage drives in the computing system. The interconnect link protocol allows direct communication between the plurality of data storage drives without participation from the host computing device.

[0009] In one example, the interconnect link protocol allows direct communication between flash translation layers configured in each of the plurality of data storage drives. The interconnect link protocol allows direct communication between buffer managers configured in each of the plurality of data storage drives. The data storage drive is a solid state drive.

[0010] Another aspect of the present disclosure includes a system for processing commands. The system includes one or more processors configured to receive an operation command in a first data storage drive disposed in a computing system. In response to the operation command, the first data storage drive directly communicates with a second data storage drive disposed in the computing system. A data storage drive in the computing system is determined to process the operation command. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram illustrating a computing system according to aspects of the present disclosure.

[0012] Figure 2 is a block diagram illustrating communications between solid-state drives (SSDs) in a computing system according to aspects of the present disclosure.

[0013] Figure 3 is a block diagram illustrating communication between flash translation layers (FTLs) in a solid state drive (SSD) according to aspects of the present disclosure.

[0014] Figure 4 is a block diagram illustrating communications between a client computing device, a host computing device, and a solid-state drive (SSD) according to aspects of the present disclosure.

[0015] Figure 5 is a data processing sequence diagram illustrating an example of data transmission between SSDs according to aspects of the present disclosure.

[0016] Figure 6 is a data processing sequence diagram illustrating another example of data transmission between SSDs according to aspects of the present disclosure.

[0017] Figure 7 is a flow diagram of example process data processing and communication in a computing system according to aspects of the present disclosure. DETAILED DESCRIPTION

[0018] The present disclosure provides an interconnection architecture that enables communication and / or data transmission between data storage drivers implemented in a computing system. In one example, the data storage driver used here is a solid-state drive (SSD). The flash translation layer (FTL) in each SSD can be configured to operate in a collaborative manner, which allows communication and / or data transmission across the data storage driver in the storage device from each SSD installed in the computing system, such as device-to-device communication or memory-to-memory communication. The FTL configured in each SSD can process write / read commands across all SSDs without deferring to the host computing device implemented in the computing system. Therefore, in an example in which an additional SSD is added to the computing system, the processing power of the FTL is also increased to adapt to the management of the additionally added SSD in the computing system, thereby providing horizontal scalability for the storage device in the computing system. This horizontal scalability enables a larger storage device to be built by increasing the number of data storage drivers implemented in the computing system without increasing the performance of the corresponding host computing device in the computing system.

[0019] Conventional interconnect architectures require a host computing device, such as a host processor, to manage communications between attached devices, such as data storage drives, in a computing system. For example, a host computing device manages the storage devices in the memory array of each data storage drive, commonly referred to as host-to-device (H2D) communication. Therefore, separate commands or requests need to be issued separately from the host computing device to the storage devices in each data storage drive for data communication. By enabling communications between data storage drives, such as device-to-device (D2D) communications, commands / requests from one data storage drive can be transmitted to another data storage drive in a collaborative manner through the configuration and settings of the FTL in the data storage drive, thereby unloading some of the computational load and memory bandwidth load required by the host computing device. Therefore, a simplified data transaction flow can be obtained with less business time and higher transmission efficiency to support emerging computing applications.

[0020] Figure 1 1 is a block diagram illustrating a computing system 100. The computing system 100 may be any type of computing device. For example, the computing system 100 may be a server system, where various devices may be in electrical communication with it over a network or through physical connections. The computing system 100 includes a host computing device 150 and a data storage drive 101. The host computing device 150 is in electrical communication with the data storage drive 101 via a single or multiple interconnects, connection buses, traces, pins, etc. In one example, Figure 1The data storage device 101 described in the figure may be a solid state drive (SSD). Note that other types of devices (such as accelerator devices) may also be implemented in the computing system 100 to improve the operating performance of the computing system 100.

[0021] The host computing device 150 and the data storage drive 101 may be implemented using the techniques described herein. In this example, the host computing device 150 may be an electronic device operably coupled to the data storage drive 101. The host computing device 150 may issue various types of commands and / or requests, such as relaying or processing requests from an I / O device 144 operated by a user to the data storage drive 101. The host computing device 150 may be in a variety of forms. As some examples, the host computing device 150 may be implemented as a CPU or a CPU used in a personal computer, a laptop computer, a server, a tablet computer, or a notebook computer, or any suitable form, together with an accelerator.

[0022] In one example, one or more I / O devices 144 may be coupled to host computing device 150 via various suitable networks (such as Ethernet-based communication protocols, WIFI connection protocols, or other suitable communication protocols). Host computing device 150 is configured to receive input requests / commands from one or more I / O devices 144.

[0023] In one example, the host computing device 150 may be connected to the data storage drive 101 via an interconnect link protocol 180 to enable data, signals, and messages to be transmitted therebetween. In some embodiments, the interconnect link protocol 180 may be used to support multiple protocols and communication of data and signals via multiple interconnect protocols, including a compute express link (CXL) protocol, a cache coherent interconnect for accelerators (CCIX) protocol, an Ethernet protocol, etc. For example, the interconnect link protocol 180 may support various interconnect protocols, including a non-coherent interconnect protocol, a coherent interconnect protocol, a memory interconnect protocol, etc.

[0024] In one example, the data storage drive 101 may be a solid-state drive that includes a controller 102 having multiple components embedded therein. The controller 102 is operably coupled to: one or more non-volatile memories, such as multiple NAND flash memories 130; and, random access memory (RAM), such as multiple DRAM memories 131, which include one or more memory dies. The memory die or chip includes a memory structure, such as a memory cell array, a control circuit, and a read / write circuit. The memory structure is addressable by a word line. The read / write circuit includes multiple memory blocks and allows for parallel reading or programming of memory cell pages. Commands and data are transmitted between the controller 102 and the one or more memory dies via communication lines in the multiple DRAM memories 131 and the multiple NAND flash memories 130.

[0025] The NAND flash memory 130 may be a two-dimensional NAND flash memory, a three-dimensional NAND flash memory. The memory structure may include a monolithic 3D memory structure, in which multiple memory levels are formed on a single substrate such as a wafer, without an intermediate substrate. The controller 102 may simulate data or signals therein from a first format to a second format, which is readable / writable for the NAND memory 130 or the DRAM memory 131.

[0026] Examples of volatile memory devices include dynamic random access memory (DRAM) or static random access memory (SRAM) devices, etc. Examples of non-volatile memory devices include resistive random access memory (ReRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, ferroelectric random access memory (FRAM) and magnetoresistive random access memory (MRAM) and other semiconductor elements capable of storing information. Each type of memory device can have a different configuration. For example, a flash memory device can be configured as a NAND or NOR configuration. The memory device can be formed by passive and / or active elements in any combination. In one example, the passive semiconductor memory element includes a ReRAM device element, which in some embodiments includes: a resistivity switching storage element, such as an antifuse or a phase change material; and, an optional steering element, such as a diode or a transistor. Active semiconductor memory elements include EEPROM and flash memory device elements, which in some embodiments include elements containing a charge storage region, such as a floating gate, conductive nanoparticles, or a charge storage dielectric material.

[0027] The controller 102 in the data storage drive 100 includes a host interface 106 having a host interface circuit configured to communicate with a host computing device 150. For example, the host interface 106 may include a non-volatile memory express specification, such as an NVMe controller or other suitable types of interface circuits to facilitate communication between the host computing device 150 and the data storage drive 101. The host interface 106 may receive various commands, such as write commands, read commands, move commands, erase commands, or other various operation commands from the host computing device 150 to perform various tasks in the data storage drive 101. For example, a write command may be sent from the host computing device 150 to request the data storage drive 101 to write data in the memory 130, 131 based on the write command. In one example, the write command may include a starting logical block address (LBA), a transfer length, and a registration ID. The registration ID in the write command may be an identifier for identifying an area where data can be written. In one example, the registration ID may be a namespace ID. The read command may request the data storage drive 101 to obtain the data stored therein. The read command may also include a starting LBA, a transfer length, and a registration ID. The registration ID in the read command can also be used to identify the area storing data to be read in the data storage drive 101. The registration ID can also be a namespace ID.

[0028] The interconnect link protocol 180 may interface with the host interface 106 to facilitate data connection or transmission therethrough.

[0029] In one example, the peripheral interface 102 may also be implemented in the data storage drive 101. The peripheral interface 102 includes a peripheral interface circuit configured to enable communication between the data storage drive or other data storage drives or other devices (such as accelerator devices) implemented in the computing system 100. For example, the peripheral interface 102 may facilitate direct data communication or direct signal transmission between external devices (such as the data storage drive 101 or other data storage drives), such as Figure 2As further shown, there is no need to defer signals back to the host computing device 150. Thus, the data storage drive 101 and other drives may communicate directly electrically with each other via the peripheral interface 102 without using the host computing device 150 as a central processing agent to further forward or transmit commands therethrough. In one example, the peripheral interface 102 may be any suitable type of interface circuitry or interface interconnect protocol to facilitate communication between the data storage drive 101 or other devices in the computing system 100. Note that in examples where the peripheral interface 102 is not present or configured in the data storage drive 101, communication between one or more data storage drives and other external devices may utilize software programming or an interconnect protocol configured in the interconnect link protocol 180 to facilitate lateral communication, such as device-to-device or drive-to-drive (D2D) communication, with no or minimal involvement of the host computing device 150.

[0030] Similarly, the peripheral interface 102 may receive various commands such as write commands, read commands, various control commands from the data storage drive 101 or other external devices. The write and read commands may be similarly configured as write and read commands from the host computing device 150 .

[0031] Multiple processors 125 may be implemented in the data storage drive 101. The multiple processors 125 are configured to control the host interface 106, the peripheral interface 102 (if present), the DRAM controller 156, and the NAND controller 155. The processor 125 performs various processes and operations by loading firmware, code, or software, etc. from the ROM 121, the NAND memory 130, and / or the DRAM 131. The memories such as the DRAM memory 131, the ROM 121, and the NAND flash memory 130 include codes, such as instruction sets, and the processor 125 is operable to execute the instruction sets to provide the functions described herein.

[0032] In one example, the processor 125 may receive commands from the host computing device 150 and / or execute such commands in the data storage drive 101. For example, the FTL 104 configured in the controller 102 may receive such commands and operate the commands for their intended purpose. In one example, the FTL 104 is configured to perform data management and block management of the NAND flash memory 130. The FTL 104 may be used to perform processes or operations such as managing write and read commands to the NAND flash memory 130 with wear leveling, maintaining a logical to physical (L2P) mapping table in the NAND flash memory 130, and providing a logical block number interface to the host computing device 150, other devices implemented in the computing system 100. Therefore, the operations or processes of the data storage drive 101 are controlled by firmware, software, or algorithms executed by the processor 125.

[0033] The processing programmed by FTL 104 may be performed by dedicated hardware, such as NAND controller 151 or processor 125 within controller 102. In one example, FTL 104 may be configured to interface with host computing device 150 through host interface 106 to handle data communications or signal transmissions. In another example, in addition to data communications or signal transmissions to or from host computing device 150, FTL 104 may be configured to interface with host computing device 150, such as Figure 2 Other data storage drives shown or other external devices (not shown) in the computing system 100 are interfaced for lateral data communication or signal transmission. The FTL 104 may be an internal module configured to be executed by the processor 125 for NAND flash memory management.

[0034] In one example, the FTL 104 may include an algorithm that converts write and read commands from the host computing device 150 into write and read commands that can be operated by the NAND flash memory 130. The NAND controller 155 is configured to control the NAND flash memory 130 under the control of the processor 125. For example, the FTL 104 may format the NAND flash memory 130 to ensure that the NAND flash memory 130 operates normally, such as mapping out bad flash memory cells, allocating spare memory cells to replace future failed cells, etc. Some parts of the spare cells may be used to store firmware to operate the NAND controller 155 and implement other features. In operation, when the host computing device 150 needs to read data from the NAND flash memory 130 or write data to the NAND flash memory 130, the host computing device 150 will communicate with the FTL 104. If the host computing device 150 provides a logical address where the data is to be read / written, the FTL 104 may convert the logical address received from the host computing device 150 into a physical address in the NAND flash memory 130. FTL 104 may also perform various memory management functions, such as, but not limited to, wear leveling and garbage collection, which are described in more detail below.

[0035] In some examples, in addition to translating write and read commands between host computing device 150 and data storage drive 101 , algorithms configured in FTL 104 may also enable communication and data transfer between data storage drives and / or other external devices implemented in computing system 100 .

[0036] The buffer manager 154 may also be configured in the processor 125. The buffer manager 154 may include an algorithm or program configured to manage a write buffer for temporarily storing write data in the DRAM 131, which may be later written to the NAND flash memory 130. The buffer manager 154 may buffer data read or written from the host computing device 150, which is configured to be written to the NAND flash memory 130 for data traffic management. The buffer manager 154 may write data from the host computing device 150 to the NAND flash memory 130 and map the memory address, and then write the data to the NAND flash memory 130 using logical block addressing (LBA). The buffer manager 154 may allocate space in a cache such as a volatile memory like the DRAM 131, so that the read command being processed has a temporary destination before returning to the host computing device 150. The buffer manager 154 may also allow write commands to be temporarily stored in a cache in the buffer manager 154 until data from the host computing device 150 can be sent to the NAND flash memory 130, or vice versa.

[0037] In one example, buffer manager 154 is also configured to communicate with other data storage drives 101 or external devices implemented in computing system 100 through execution of processor 125. Thus, content from DRAM 131 can be transferred to other DRAM or memory in computing system 100 to improve utilization of storage capacity in computing system 100 with minimal involvement from host computing device 150.

[0038] DRAM controller 156 is configured to control DRAM memory 131 under the control of processor 125. Programs and algorithms in buffer manager 154 and flash translation layer 104 may be executed and implemented by processor 125 to control read and write commands in data storage drive 101 for data communication or signal transmission in computing system 100.

[0039] Figure 2An example of a computing system 200 having a plurality of data storage drives 101, 202a, 202b, 202c implemented therein is depicted. The data storage drives 202a, 202b, 202c, which are configured similarly or identically to the data storage drive 101, may also be solid-state drives (SSDs). The plurality of data storage drives 101, 202a, 202b, 202c may be in direct electrical communication without or with less involvement of the host computing device 150. As described above, the flash translation layer (FTL) 104 in each of the plurality of data storage drives 101, 202a, 202b, 202c may be configured to not only communicate electrically with the host computing device 150, but also to perform direct electrical communication between the data storage drives 101, 202a, 202b, 202c without intervention or involvement from the host computing device 150. For example, the FTL 104 may have an algorithm configured to allow direct transmission of requests and commands between the data storage drivers 101, 202a, 202b, 202c. The peripheral interface 102 in each of the data storage drivers 101, 202a, 202b, 202c may transmit a signal or a read or write command from the FTL 104 or the buffer manager 154 to the corresponding FTL or the corresponding buffer manager in each data storage driver 101, 202a, 202b, 202c. The interconnect link protocol connected and programmed between the data storage drivers 101, 202a, 202b, 202c may implement data, signal or command transmission between the data storage drivers 101, 202a, 202b, 202c. In some examples, the interconnect link protocol may be operable to support multiple protocols and communication of data and signals via multiple interconnect protocols, including a computing express link (CXL) protocol, a cache coherent interconnect (CCIX) protocol for accelerators, an Ethernet protocol, and the like. The interconnect link protocol can be docked with the peripheral interface 102 for data transmission or connection. Note that the interconnect protocol is configured in a manner that allows linear or lateral communication between the data storage drives 101, 202a, 202b, 202c without the participation or relay from the host computing device 150, thereby unloading some of the computing load from the host computing device 150. The interconnect link protocol can support a variety of interconnect protocols, including non-coherent interconnect protocols, coherent interconnect protocols, memory interconnect protocols, etc. In addition, each of the data storage drives 101, 202a, 202b, 202c can also be physically connected via a single or multiple interconnects 180, 209 (such as, connecting buses, interconnect traces, pins, etc.) to facilitate data transmission and / or signal communication between them. In some examples, the physical connection 209 can be a switching structure.

[0040] In an example where the peripheral interface 102 is not present in the data storage drive 101, 202a, 202b, 202c, data communication or signal transmission between the data storage drives 101, 202a, 202b, 202c can utilize the interconnection link 180 configured in the computing system 200, which interfaces with the host interface 106. Therefore, vertical communication (such as host to device (H2D) communication) together with horizontal communication (such as device to device (D2D) communication or referred to as horizontal or linear communication) can be integrated and enabled by the interconnection link protocol 180. For example, data communication and / or signal transmission can utilize the protocol configured in the interconnection link protocol 180 to facilitate horizontal or horizontal communication in the data storage drive 101, 202a, 202b, 202c. As described above, the interconnection link protocol 180 is operable to support multiple protocols and communication of data and signals via multiple interconnection protocols (including the Compute Express Link (CXL) protocol, the Cache Coherent Interconnect for Accelerators (CCIX) protocol, the Ethernet protocol, etc.). These interconnect protocols may be programmed or configured to allow horizontal, lateral or linear communication, such as D2D communication, between the data storage drives 101 , 202a , 202b , 202c .

[0041] The FTL 104 or buffer manager 154 programmed in each data storage drive 101, 202a, 202b, 202c may perform processing in coordination with other FTLs 104 or buffer managers 154 in other data storage drives 101, 202a, 202b, 202c in the computing system 200. Thus, the use, consumption, and capacity of the NAND flash memory 130 and the DRAM 131 may be accessible and readable / writable by the processor 125 from any of the data storage drives 101, 202a, 202b, 202c implemented and available in the computing system 200. Thus, when a data write command is initiated, the FTL 104 or buffer manager 154 from the data storage drives 101, 202a, 202b, 202c may coordinately to efficiently locate storage space for the data write request across the available NAND flash memory 130 and / or DRAM 131 in the data storage drives 101, 202a, 202b, 202c implemented in the computing system 200. The write command processing by the FTL 104 or buffer manager 154 may be sliced ​​or divided across the available processors 125 in the data storage drives 101, 202a, 202b, 202c to enable parallel processing across the processors 125 of the data storage drives 101, 202a, 202b, 202c. Thus, the workload from the FTL 104 or buffer manager 154 may be divided and transmitted across the D2D network to the available processors 125, so that the processors 125 from each of the data storage drives 101, 202a, 202b, 202c may share and process a portion of the workload. Thus, the workload sharded or divided across the data storage drives 101, 202a, 202b, 202c to different processors may simultaneously process commands assigned to the FTL 104 or buffer manager 154 without involvement from the host computing device 150. Thus, by doing so, the processing performed by each FTL 104 or buffer manager 154 in each of the data storage drives 101, 202a, 202b, 202c may not need to be deferred back to the host computing device 150 for further dispatch, thereby reducing the computational load and memory bandwidth load from the host computing device 150. Furthermore, the capacity of the storage device may be increased by simply adding additional data storage drives 101, 202a, 202b, 202c implemented in the computing system 200 and increasing the scalability of storage applications across the data storage drives 101, 202a, 202b, 202c without having to replace the host computing device 150 with higher performance requirements.

[0042] Figure 3A block diagram illustrating the communication between FTLs 104, 350a-c in SSDs 101, 202a-c according to aspects of the present disclosure is depicted. Note that other components (such as processors, DRAM, or buffer managers, etc.) are not Figure 3 , are removed for ease of illustration and description. As described above, the FTL 104, 350a-c in each data storage drive 101, 202a-c is configured to communicate electrically with each other when executing a process performed by a processor. Note that the buffer manager in each data storage drive 101, 202a-c may also be similarly configured. The FTL 104, 350a-c may include a logical to physical mapping block 302, 352a-c, a garbage collection block 304, 354a-354c, and a wear leveling block 306, 356a-c, etc. The logical to physical mapping block 302, 352a-c provides a mapping algorithm that can manage the logical to physical mapping of data within a physical memory such as a NAND flash memory 130, 340a-c. FTL 104, 350a-c may be responsible for converting logical block addresses provided by a user (such as write signals transmitted from I / O device 144 through host computing device 150 to any of SSDs 101, 202a-c) into physical block addresses in NAND flash memory 130, 340a-c.

[0043] Garbage collection (GC) blocks 304, 354a-c also provide GC algorithms that are configured to move data back and forth in NAND flash memory 130, 340a-c to ensure that the program / erase (P / E) cycle of each physical memory block can be unified. GC blocks 304, 354a-c can access allocation tables configured in each data storage driver. The allocation table can include at least global drive wear level and / or global drive attributes. The access of GC blocks 304, 354a-c to global drive wear level and / or global drive attributes can ensure that the P / E cycle is consistent across all memory blocks in each of the data storage drivers and across all data storage drivers. The GC algorithm in FTL 104, 350a-c can also be responsible for reclaiming physical storage blocks containing invalid data in the following manner: copying their valid data to new blocks and erasing them in GC blocks 304, 354a-c. The GC algorithm in the FTL 104, 350a-c may also move only valid data pages to a new block in the data storage drive or to another data storage drive after a block is full, so that full blocks can be erased and reused.

[0044] In addition, over time, the NAND flash memory 130, 340a-c may wear out. Therefore, the wear leveling blocks 306, 356a-c may provide a wear leveling (WL) algorithm that ensures that the physical memory wears out in a consistent manner by writing or copying data to the appropriate locations in the physical memory. The WL algorithm may evenly distribute writes to avoid wearing out specific memory blocks that would otherwise be repeatedly written to.

[0045] By configuring algorithms in the FTL 104, 350a-c to allow direct communication between the SSDs 101, 202a-c, the GC, logical to physical mapping, and wear leveling functions performed in the NAND flash memory 130, 340a-c can be operated jointly and cooperatively. For example, the coordinated operation of the drives 101, 202a-c can balance wear across the entire group of drives 101, 202a-c in a relatively consistent manner to avoid the possibility of one particular drive becoming a wear hotspot. This benefit can also be maintained and realized while the drives continue to provide the same total capacity and the same static allocation of logical block address space as the host computing device 150 is configured to operate.

[0046] Therefore, NAND flash memories 130, 340a-c located at different SSDs 101, 202a-c can be managed by any of the FTLs 104, 350a-c in the SSDs 101, 202a-c. For example, the processes and / or functions performed by the GC blocks 304, 354a-c, the logical-to-physical mapping blocks 302, 352a-c, and the wear leveling blocks 306, 356a-c can be coordinated at the storage device level between the SSDs 101, 202a-c with little or no involvement of the host computing device. In addition, the degree of wear between the NAND flash memories 130, 340a-c in the SSD 101, 202a-c can be detected in a common manner by the FTL 104, 350a-c, so that data across the NAND flash memories 130, 340a-c can be remapped to avoid certain areas or certain specific NAND flash memories 130, 340a-c wearing faster than other areas.

[0047] In one example, a variety of different types of memory devices can be used to implement the physical memory. In one example described herein, a flash memory device such as a NAND flash memory device 130 or other types of non-volatile memory devices can be used to implement the physical memory. In some embodiments, a volatile memory device such as a random access memory device can be used for the physical memory. Note that other types of physical memory are also possible, and other types of memory devices can be used to implement what is described herein.

[0048] In addition, the processor 125 can generate namespaces that define logical blocks of data stored in the NAND flash memories 130, 340a-c. The generation of namespaces also consumes resources of the NAND flash memories 130, 340a-c. Therefore, a single SSD may only have a certain limited capacity to allow a certain number of namespaces to be generated. In one example, 8 namespaces can be supported in a single SSD. Using device-to-device communication that allows direct communication between the FTLs 104, 350a-c, the namespaces assigned or generated in each SSD can be combined, shared, or integrated using a programmed algorithm, thereby presenting an 8xN namespace of additional resources available for use between the SSDs in the computing system, for example, N is the number of SSDs implemented in the computing system.

[0049] Figure 4 1 is a block diagram illustrating an example of communication between a client computing device, a host computing device, and a solid state drive (SSD) according to aspects of the present disclosure. One or more client computing devices 360a, 360b, 360c, 360d (similar to any suitable type of I / O device 144 described above) can be used to provide commands to the host computing device 150. The commands can be transmitted to the host computing device 150 via various suitable networks 361. A processor (such as a CPU 365) can be implemented in the host computing device 150, which is configured to service read and write commands issued from the client computing devices 360a-360d. The CPU 365 can allocate a logical address space to each drive 101, 202a, 202b, 202c via the interconnect link protocol 180 or other suitable communication protocol. Therefore, each read or write command transmitted from the client computing device 360a-360d can reach the specified drive to service the logical address. The CPU 365 communicates with the drives 101, 202a, 202b, 202c via the interconnect link protocol 180 for read and write commands.

[0050] By utilizing drive-to-drive communication, the interconnection link protocol 180 and / or the horizontal interconnection link protocol can allow communication between the drivers 101, 202a, 202b, 202c. The drivers 101, 202a, 202b, 202c communicate with each other so that commands from the CPU 365, address mappings, and wear statistics of their own drivers can be transmitted between them. The drivers 101, 202a, 202b, 202c can transmit data to each other through the interconnection link protocol 180 and / or the horizontal interconnection link protocol to achieve drive-to-drive (D2D) communication. The drivers 101, 202a, 202b, 202c can also instruct other drivers 101, 202a, 202b, 202c to transfer data to the CPU 365 at the provided address mapping.

[0051] In one example, each driver 101, 202a, 202b, 202c is configured to maintain each FTL that allows the allocated logical blocks to be mapped to remote drivers or other drivers in the system. The FTL can also maintain the physical addresses of the logical blocks held by the drivers 101, 202a, 202b, 202c. Each driver 101, 202a, 202b, 202c may include an allocation table, such as at least a global driver attribute table and / or a global driver wear statistics table, which displays the attributes or wear levels of each driver in the storage system. Since each driver 101, 202a, 202b, 202c may have different attributes, such as different storage capacities and / or wear capabilities, such a global driver attribute table and / or wear statistics table can provide relevant driver attributes and information to each driver 101, 202a, 202b, 202c allocated in the system. For example, a quad-level cell (QLC) driver may have a higher capacity but a lower wear capability than a triple-level cell (TLC) driver.

[0052] Thus, when the CPU 365 issues a write command to one of the drives 101, 202a, 202b, 202c (such as drive 101), the selected drive 101 can check the global wear statistics and global drive attributes in the table and determine whether the write command should be transmitted to a more appropriate remote drive or other suitable drive in the system. In the event that the selected drive determines to accept the write command, the previously written block in the drive can be transmitted or passed to another remote or local drive that is more suitable to store the previously written block.

[0053] In contrast, when the CPU 365 issues a read command to the drive, the drive can check its FTL to locate the drive where the data resides. Once the data is located, the read command can be transmitted to the drive where the data resides. In some examples, the remote drive will directly transmit the data to the CPU 365.

[0054] In addition, drive-to-drive communication can also utilize a collaborative storage system with heterogeneous drives and automatic tiering to provide better capacity and / or performance operating points and reduced operating costs. For example, the collaborative drive can perform automatic tiering based on the type of data characteristics received and the drives available in the collaborative storage system. Like this, each type of data saved to the drive may have matching attributes. For example, the automatic tiering may include a group of fast drives, which may be a fast tier, also known as a high tier. In contrast, a group of slow drives may be a slow tier, also known as a low tier. Therefore, different tiers from the automatic tiering function can allow different attributes of data stored in different drives to achieve desired operating performance at a desired price range. Data stored in the fast tier is generally referred to as hot data, while data stored in the slow tier is generally referred to as cold data. Therefore, by utilizing collaborative storage operations, cold data can be configured to be stored in a quad-level cell (QLC) drive (such as a lower tier drive), while hot data can be configured to be stored in a relatively high-endurance multi-level cell (MLC) drive (such as a higher tier drive). Frequently read data can be configured to be stored in a low-latency drive. Therefore, by utilizing a collaborative storage system, good data matching and distribution management can be obtained to achieve high operating performance and keep operating costs low.

[0055] Figure 5 Depicted is a data processing sequence diagram illustrating an example data transfer between data storage drives utilizing device-to-device communication according to aspects of the present disclosure. Figure 5 In the example shown in , an input write command may be initiated by a user utilizing I / O device 144. The write command may then be transmitted to host computing device 150 for processing, as shown by signal flow path 402. Note that commands that may be transmitted between components in a computing system may include data write commands, data read commands, data write / program commands, and data erase commands, as well as other types of data commands / requests, such as, for example, move commands and / or copy commands.

[0056] The processor in computing device 150 may determine to which data storage drive among the data storage drives available in the computing system to transmit the write command for storage. Figure 5In the example shown, the processor in the computing device 150 can also transmit a write command to the device 2, as shown in the signal flow path 404, in order to store the write command in the device 2 memory. However, the device 2 memory may not be ready to receive the write command for any reason, such as a memory unit failure, etc. Therefore, instead of postponing the write command back to the host computing device 150 for further dispatch or instruction (as shown in the signal flow path 411), the write command can be transmitted to other data storage drivers, such as the device 3 (as shown in the signal flow path 406). The device-to-device communication provided by the FTL in each device can communicate together in a collaborative manner to determine the appropriate data storage driver for processing the write command. Therefore, the device-to-device communication enables direct communication between the driver 1, the driver 2 and the driver 3. Therefore, the write command can be transmitted to a suitable driver, such as the driver 3, to store the write command in the device 3 memory, as shown in the signal flow path 408.

[0057] Figure 6 5 is a data processing sequence diagram of another example process for processing write commands between solid-state drives in a computing system utilizing D2D communication according to aspects of the present disclosure. For example, when a write command (such as a command to rewrite data from a defective block in a first drive to a good block in a second drive) is initiated from the FTL in drive 2, instead of deferring the write command to the host computing device 150, the write command can be directly transmitted to the FTL in drive 3 through direct communication between them to determine the appropriate drive for processing the write command (as shown in signal flow path 506). The FTL in drive 3 can then process the write command in drive 3, such as rewriting data to a good block in the memory of drive 3, as shown in signal flow path 508, without deferring the write command to the host computing device for decision, as shown in signal flow path 510.

[0058] Figure 7 600 is a flow chart of an example process 600 for processing write commands between solid-state drives in a computing system utilizing device-to-device communication according to aspects of the present disclosure. Such methods may be performed using any of the above-described systems, modifications thereof, or a variety of systems having different configurations. It should be understood that the operations involved in the following methods need not be performed in the exact order described. Instead, various operations may be handled in a different order or simultaneously, and operations may be added or omitted.

[0059] Process 600 begins at block 602 by receiving a write command in a first driver of a data storage drive implemented in a computing system. When the write command is received in the first driver, the FTL in the first driver may process the write command and determine a memory block for storing data from the write command.

[0060] In one example, at block 604, it may be determined that data from a write command is written to a first drive memory in a first drive.

[0061] However, in examples where data from the write command fails to be written to the first drive memory or is determined to be moved to other memory blocks for storage, the write command may then be further redirected back to the first drive, as indicated by arrow 627, and the first drive may retransmit the write command to the second drive, as indicated by block 614. The FTL in the second drive may then process the write command and determine a storage block in the second drive memory for data storage, as indicated by block 616.

[0062] However, in an example where the data from the write command fails to be written to the second drive memory or is determined to be moved to other memory blocks for storage, the write command may then be further redirected back to the first drive, as indicated by arrow 620, and the first drive may then communicate with the third drive, as indicated by arrow 624, to request data storage, as indicated by block 632. Alternatively, the write command may then be further redirected back to the second drive, as indicated by arrow 628, and the second drive may retransmit the write command to the third drive, as indicated by arrow 626, for data storage, as indicated by block 632. Alternatively, the communication between the first, second, and third FTL drivers configured in each drive may communicate simultaneously to determine the appropriate memory blocks available for processing and storing the write command.

[0063] The present disclosure provides an interconnect protocol that enables direct communication and / or data transfer between data storage drives implemented in a computing system with minimal involvement from a host computing device in the computing system. A flash translation layer (FTL) in each data storage drive can operate in a collaborative manner that allows direct communication and / or data transfer, such as D2D communication or memory-to-memory communication, across data storage drives in a storage device in the computing system. Therefore, D2D communication enables larger storage devices to be built at the drive level without increasing the performance of a host computing device implemented in the computing system. Therefore, the flexibility of expanding storage devices in a computing system becomes simple and less complex.

[0064] Although a process may be described as including multiple operations, this does not mean that all or any of the operations are preferred, essential, or necessary. Various other embodiments may include other processes that omit some or all of the described operations. Although a process may be described alone or without reference to other products or processes, in some examples, a process may interact with other products or processes. Such interaction may be provided to enhance the flexibility or desirability of the process.

[0065] Aspects of the present disclosure may be executed on a single device or may be executed on multiple devices. For example, program modules including one or more components described herein may be located in different devices and may each execute one or more aspects of the present disclosure. The terms "one" or "an" used in the present disclosure may include one or more items, unless otherwise specifically stated. In addition, unless otherwise specifically stated, the phrase "based on" is intended to mean "based at least in part on".

[0066] Unless otherwise stated, the above alternative examples are not mutually exclusive, but can be implemented in various combinations to obtain unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the examples should be given by way of illustration rather than by limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein and the terms expressed as "such as" and "including" should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, these examples are intended to illustrate only one of many possible examples. In addition, the same figure numbers in different figures may identify the same or similar elements.

[0067] Many examples are described in this application and are given for illustrative purposes only. The examples described are not, nor are they intended to be, limiting in any sense. One of ordinary skill in the art will recognize that the disclosed subject matter can be practiced through various modifications and variations, such as structural, logical, software, and electrical modifications. It should be understood that unless otherwise expressly stated, the features described are not limited to use in one or more specific examples or drawings, with reference to which the described features are described.

[0068] The above discussion is provided in the context of specific applications and their requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the disclosed technology. Therefore, the disclosed technology is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

Claims

1. A method for processing commands in a computing system, comprising: receiving, by one or more processors, an operation command in a first data storage drive disposed in the computing system; In response to the operation command, the first data storage drive directly communicates with a second data storage drive disposed in the computing system; determining, by the one or more processors, based on the communication, a data storage drive in the computing system to handle a workload associated with the received operation command, wherein the workload is divided across the determined data storage drives; as well as The divided workload is processed by the determined data storage drives.

2. The method according to claim 1, wherein: Determining the data storage drive is further based on a type of the operating command.

3. The method according to claim 2, wherein: When the operation command is a read command, the determination of the data storage driver indicates where data resides in the first data storage driver and / or the second data storage driver based on a flash translation layer.

4. The method according to claim 2, wherein: When the operation command is a write command, the determination of the data storage drive includes at least one of a global drive attribute and a global drive wear statistic based on an allocation table.

5. The method according to claim 4, wherein: The global drive attributes include at least one of capacity and wear-out capability.

6. The method according to claim 1, wherein: Communicating directly with the second data storage drive further includes communicating directly without communicating with a host computing device in the computing system.

7. The method according to claim 1, wherein: The first and second data storage drives are solid state drives.

8. The method according to claim 1, wherein: The operation command is a data write command, a data read command, a data program command or a data erase command.

9. The method according to claim 1, wherein: Receiving the operation command in the first data storage drive further comprises: The operation command is processed by the one or more processors through a first buffer manager configured in the first data storage drive.

10. The method according to claim 9, further comprising: Direct communication is performed by the one or more processors from the first buffer manager to a second buffer manager configured in the second data storage drive.

11. The method according to claim 1, wherein: Data blocks are defined in a plurality of non-volatile memories or a plurality of volatile memories arranged in the first data storage drive or the second data storage drive.

12. The method according to claim 11, wherein: The plurality of nonvolatile memories are a plurality of NAND flash memories.

13. The method according to claim 11, wherein: The plurality of volatile memories are a plurality of DRAMs.

14. The method according to claim 1, wherein: The direct communication between the first data storage drive and the second data storage drive is transmitted through an interconnect link protocol defined between the first data storage drive and the second data storage drive.

15. The method according to claim 14, wherein: The interconnect link protocol enables direct communication between the first data storage drive and the second data storage drive without requiring communication with a host computing device in the computing system.

16. A computing system comprising: host computing device; a plurality of data storage drives in communication with the host computing device; as well as An interconnect link protocol connected to the plurality of data storage drives in the computing system, wherein the interconnect link protocol allows direct communication between the plurality of data storage drives without involvement from the host computing device, wherein a workload associated with an operation command received by one of the plurality of data storage drives is divided across the data storage drives in the plurality of data storage drives for processing.

17. The computing system of claim 16, wherein: The interconnect link protocol allows direct communication between flash translation layers configured in each of the plurality of data storage drives.

18. The computing system of claim 16, wherein: The interconnect link protocol allows direct communication between buffer managers configured in each of the plurality of data storage drives.

19. The computing system of claim 16, wherein: The plurality of data storage drives are solid state drives.

20. A system for processing commands, the system comprising one or more processors, the one or more processors being configured to: receiving an operation command in a first data storage drive disposed in a computing system; in response to the operation command, communicating directly from the first data storage drive to a second data storage drive disposed in the computing system; as well as determining a data storage drive in the computing system to handle a workload associated with the received operational command, wherein the workload is divided across the determined data storage drives; as well as The divided workload is processed by the determined data storage drives.