Storage device and method of operating the same

Abstract

A storage device includes a buffer memory configured to temporarily store data, a plurality of non-volatile memory devices, a storage controller circuit configured to generate buffer memory status information by monitoring a status of the buffer memory, and operate in a congestion control mode of setting a buffer memory data transfer grant of the non-volatile memory based on the generated buffer memory status information, and a first interface circuit configured to communicate with the storage controller circuit and the plurality of non-volatile memory devices, wherein the first interface circuit is connected to a network based on an Ethernet interface.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10-2020-0174729, filed on December 14, 2020, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] At least some exemplary embodiments of the present invention relate to storage systems, and more specifically, to storage systems including congestion control functionality using tokens. Background Technology

[0004] A prime example of a high-capacity flash-based storage device is the solid-state drive (SSD). With the explosive growth in demand for SSDs, their applications have become increasingly segmented. For instance, SSDs can be categorized as server SSDs, client SSDs, data center SSDs, and so on. The SSD interface should provide optimal speed and reliability for each application. To meet these requirements, Non-Volatile Memory High-Speed ​​(NVMe) based on Serial Advanced Technology Attachment (SATA), Serial Attached Small Computer System Interface (SCSI) (SAS), and Peripheral Component Interconnect High Speed ​​(PCIe) has seen rapid research and adoption.

[0005] Recently, SSD interfaces for providing scalability in systems such as high-capacity data centers have seen rapid development. In particular, NVMe over Fabrics (NVMe-oF), as a standard for installing SSDs on network fabrics such as Ethernet switches, has been rapidly explored. NVMe-oF supports the NVMe storage protocol through a wide range of storage networking infrastructures, such as Ethernet, Fibre Channel, InfiniBand, etc. Summary of the Invention

[0006] According to at least some exemplary embodiments of the present invention, a storage device includes a buffer memory configured to temporarily store data; a plurality of non-volatile memory devices; a storage controller circuit configured to generate buffer memory status information by monitoring the status of the buffer memory, and to operate in a congestion control mode that sets buffer memory data transmission authorization for the non-volatile memory based on the generated buffer memory status information; and a first interface circuit configured to communicate with the storage controller circuit and the plurality of non-volatile memory devices, wherein the first interface circuit is connected to a network via an Ethernet interface.

[0007] According to at least some exemplary embodiments of the present invention, a method of operating a storage device includes: generating buffer memory state information by monitoring the state of a buffer memory; setting a data transfer mode for a plurality of non-volatile memory devices based on the generated buffer memory state information; and controlling the plurality of non-volatile memory devices based on the set data transfer mode, wherein the data transfer mode includes a congestion control mode that limits the amount of data transfer when the utilization rate of the buffer memory is greater than a preset reference value.

[0008] According to at least some exemplary embodiments of the present invention, a non-transitory computer-readable recording medium has computer-readable instructions recorded thereon, which, when executed by one or more processors, cause the one or more processors to perform operations, said operations including: generating buffer memory state information by monitoring the state of a buffer memory; setting a data transfer mode for a plurality of non-volatile memory devices based on the generated buffer memory state information; and controlling the plurality of non-volatile memory devices based on the set data transfer mode, wherein the data transfer mode includes a congestion control mode that limits the amount of data transfer when the utilization rate of the buffer memory is greater than a preset reference value. Attached Figure Description

[0009] The above and other features and advantages of the inventive concept will become clearer from the detailed description of exemplary embodiments of the inventive concept with reference to the accompanying drawings. The drawings are intended to depict exemplary embodiments of the inventive concept and should not be construed as limiting the scope contemplated by the claims. Unless explicitly stated otherwise, the drawings should not be considered as drawn to scale.

[0010] Figure 1 This is a block diagram of a storage system according to at least one exemplary embodiment of the present invention;

[0011] Figure 2 This is a block diagram of a storage device according to at least one exemplary embodiment of the present invention;

[0012] Figure 3 This is a flowchart illustrating the operation of a storage device according to at least one exemplary embodiment of the present invention.

[0013] Figure 4 This is a signaling diagram of the operation of a storage device according to at least one exemplary embodiment of the present invention;

[0014] Figure 5 This is a signaling diagram of a storage device in congestion control mode according to at least one example embodiment of the present invention.

[0015] Figure 6This is a flowchart illustrating the operation of managing data tokens in a storage device according to at least one exemplary embodiment of the present invention.

[0016] Figure 7 It is a table of congestion control signals for a storage device according to at least one example embodiment of the present invention;

[0017] Figure 8 It is a priority table of a storage device according to at least one example embodiment of the present invention;

[0018] Figure 9 These are diagrams illustrating a congestion control method for a storage device according to at least one exemplary embodiment of the present invention; and

[0019] Figure 10 This is a block diagram of a data center that applies a storage device based on at least one example embodiment of the concept according to the present invention. Detailed Implementation

[0020] As is customary in the field of inventive concepts, embodiments are described and illustrated in the accompanying drawings in the form of functional blocks, units, and / or modules. Those skilled in the art will understand that these blocks, units, and / or modules are physically implemented by electronic (or optical) circuitry (such as logic circuits, discrete components, microprocessors, hardwired circuits, memory elements, wiring connections, etc.), which can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. Where blocks, units, and / or modules are implemented by microprocessors or the like, they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and they can optionally be driven by firmware and / or software. Alternatively, each block, unit, and / or module can be implemented by dedicated hardware, or as a combination of dedicated hardware performing some functions and processors performing other functions (e.g., one or more programmed microprocessors and associated circuitry). Furthermore, without departing from the scope of the inventive concept, each block, unit, and / or module of the embodiments can be physically separated into two or more interactive and discrete blocks, units, and / or modules. Furthermore, without departing from the scope of the inventive concept, the blocks, units, and / or modules of the embodiments can be physically combined into more complex blocks, units, and / or modules.

[0021] Figure 1 This is a block diagram of a storage system 10 according to at least one exemplary embodiment of the present invention.

[0022] refer to Figure 1Storage system 10 may include host 100, network 300, and storage device 200. Host 100 and storage device 200 can exchange data through network 300. Data may include commands and may be referred to as packets, frames, messages, etc. Data sent from host 100 may be referred to as command data, and data sent from storage device 200 may be referred to as response data. Network 300 may include a network fabric, such as Fibre Channel (FC), Ethernet, or wireless bandwidth.

[0023] The host 100 and storage device 200 may include at least one network interface (NIF) configured to connect to the network 300. The NIF may be referred to as a network interface card, network interface controller, local area network (LAN) card, Ethernet adapter, Ethernet card, etc.

[0024] Host 100 may include host network interface (h_NIF) 130, host controller 110, and memory 120. Storage device 200 may include storage network interface 230, buffer memory 240, storage controller 210, bus 220, and non-volatile memory (NVM) 250 including multiple NVM devices 251, 252, 253, ...

[0025] h_NIF 130 can provide data to or receive data from storage device 200 via network 300.

[0026] Host controller 110 can generate multiple commands. These commands can be write commands, read commands, or erase commands, but are not limited to these. Host controller 110 can sequentially provide the generated commands to storage device 200. Storage device 200 can sequentially process the received commands and provide information about the processed commands to host 100. This information about the processed commands may include command identification information. Host controller 110 can manage operations that store data in a buffer area (e.g., write data) in NVM 250 or data in NVM 250 (e.g., read data) in a buffer area. In this specification, NVM 250 may also be referred to as storage device 250.

[0027] According to at least some example embodiments, host 100 and / or host controller 110 may include processing circuitry systems, such as hardware including logic circuitry, hardware / software combinations executing software, or combinations thereof. More specifically, the processing circuitry systems may include, but are not limited to, one or more of a central processing unit (CPU), processor core, arithmetic logic unit (ALU), digital signal processor, microprocessor, microcomputer, field-programmable gate array (FPGA), programmable logic unit, application-specific integrated circuit (ASIC), etc. According to at least some example embodiments, host controller 110 may be implemented by one or more circuits or circuitry systems. Therefore, in this specification, host controller 110 may also be referred to as host controller circuitry 110.

[0028] The memory 120 can be used as a buffer memory configured to temporarily store data to be sent to or from the storage device 200.

[0029] According to embodiments, the host controller 110 and memory 120 may be implemented by separate semiconductor chips. Alternatively, in some embodiments, the host controller 110 and memory 120 may be integrated into the same semiconductor chip. For example, the host controller 110 may be any one of a plurality of modules included in an application processor, and the application processor may be implemented by a system-on-a-chip (SoC). Furthermore, the memory 120 may be embedded memory included in the application processor, or an NVM or memory module external to the application processor.

[0030] The data may include identifier (ID) information of h_NIF 130 included in host 100, and network 300 may transmit data to h_NIF 130 by referencing the ID information. Furthermore, by referencing the ID information, a host may send data to and receive data from multiple storage devices, and multiple hosts may use a single storage device.

[0031] Storage controller 210 can control buffer memory 240, which is configured to temporarily store data of NVM 250.

[0032] Buffer memory 240 can temporarily store (buffer) data to be written to or read from NVM 250. In some embodiments, buffer memory 240 may include dynamic random access memory (DRAM). Furthermore, buffer memory 240 may store metadata, which may indicate user data or data generated by storage controller 210 to manage NVM 250.

[0033] Storage controller 210 can be connected to NVM 250 via bus 220. Bus 220 may include, but is not limited to, one of the following interfaces: Ethernet interface, Peripheral Component Interconnect High Speed ​​(PCIe) interface, Small Computer System Interface (SCSI), Serial Advanced Technology Attachment (SATA) interface, or Integrated Drive Electronics (IDE) interface.

[0034] Each of the plurality of NVM devices 251, 252, 253, ... can perform an operation corresponding to a command received from the storage controller 210. Each of the plurality of NVM devices 251, 252, 253, ... can perform the operation and then provide operation completion information to the storage controller 210. Each of the plurality of NVM devices 251, 252, 253, ... can include at least one of flash memory, phase-change random access memory (PRAM), resistive random access memory (ReRAM), DRAM, and static random access memory (SRAM).

[0035] Figure 2 This is a block diagram of a storage device 200 according to at least one exemplary embodiment of the present invention.

[0036] Storage device 200 may include storage controller 210 and NVM 250. Storage device 200 may include storage media configured to store data in response to requests from host 100. For example, storage device 200 may include at least one of solid-state drive (SSD), embedded memory, and removable external memory. When storage device 200 is an SSD, it may conform to the Non-Volatile Memory High Speed ​​(NVMe) standard. When storage device 200 is embedded memory or external memory, it may conform to Universal Flash Memory (UFS) or Embedded Multimedia Card (eMMC) standards. Storage device 200 may generate and send packets according to the appropriate standard protocol.

[0037] When the NVM 250 in storage device 200 includes flash memory, the flash memory may include a two-dimensional (2D) NAND memory array or a three-dimensional (3D) (or vertical) NAND memory array. As another example, storage device 200 may include various other types of NVM. For example, magnetic RAM (MRAM), spin-transfer torque MRAM, conductive bridged RAM (CBRAM), ferroelectric RAM (FeRAM), PRAM, ReRAM, and various other types of memory may be applied to storage device 200.

[0038] Storage controller 210 may include storage interface 211, memory interface 212, and central processing unit (CPU) 213. Furthermore, storage controller 210 may also include flash translation layer (FTL) 214, packet manager 215, buffer memory 216, congestion control unit (CCU) 217, and monitoring unit (MU) 218. Storage controller 210 may also include working memory (not shown) in which FTL 214 is loaded, and CPU 213 can control data write and read operations on NVM 250 by executing FTL 214.

[0039] According to at least some example embodiments, any one or all of the Flash Translation Layer (FTL) 214, Packet Manager 215, Congestion Control Unit (CCU) 217, and Monitoring Unit (MU) 218 ​​of the storage controller 210 may be implemented by a processing circuitry system (such as hardware including logic circuitry, a hardware / software combination executing software, or a combination thereof). More specifically, the processing circuitry system may include, but is not limited to, one or more of a Central Processing Unit (CPU), processor core, Arithmetic Logic Unit (ALU), Digital Signal Processor, Microprocessor, Microcomputer, Field Programmable Gate Array (FPGA), Programmable Logic Unit, Application-Specific Integrated Circuit (ASIC), etc. Therefore, in this specification, FTL 214, Packet Manager 215, CCU 217, MU 218, and storage controller 210 may also be referred to as FTL circuitry 214, Packet Manager circuitry 215, CCU circuitry 217, MU circuitry 218, and storage controller circuitry 210.

[0040] According to at least some example embodiments, buffer memory 240, NVM 250, and / or each NVM device included in NVM 250 (e.g., NVM devices 251-253) may include processing circuitry, such as hardware including logic circuitry, hardware / software combination executing software, or combinations thereof. More specifically, for example, the processing circuitry may include, but is not limited to, one or more of a central processing unit (CPU), processor core, arithmetic logic unit (ALU), digital signal processor, microprocessor, microcomputer, field-programmable gate array (FPGA), programmable logic unit, application-specific integrated circuit (ASIC), etc.

[0041] Storage interface 211 can send packets to and receive packets from host 100. Packets sent from host 100 to storage interface 211 may include commands, data to be written to NVM 250, etc., and packets sent from storage interface 211 to host 100 may include responses to commands, data read from NVM 250, etc. Storage interface 212 can send data to NVM 250 to be written to NVM 250, or receive data read from NVM 250. Storage interface 212 can be implemented in accordance with standards such as Toggle or Open NAND Flash Interface (ONFI).

[0042] The FTL 214 can perform several functions, such as address mapping, wear-leveling, and garbage collection. Address mapping translates logical addresses received from host 100 into physical addresses for actual data storage in the NVM 250. Wear-leveling is a technique that prevents excessive degradation of specific blocks by ensuring even use of blocks within the NVM 250, and can be implemented through firmware techniques such as balancing the erase counts of physical blocks. Garbage collection is a technique that ensures available capacity in the NVM 250 by copying valid data from a block to a new block and then erasing the existing block.

[0043] Packet manager 215 can generate packets according to the protocol of the interface consistent with host 100, or parse various information from packets received from host 100. Furthermore, buffer memory 216 can temporarily store data to be written to or read from NVM 250. Buffer memory 216 may be included in or outside of storage controller 210.

[0044] The CCU 217 can control the data transmission of the NVM 250 based on the state of the buffer memory 216. The CCU 217 can grant the NVM 250 access to the buffer memory 216, adjust the data transmission volume and bandwidth, and schedule the data transmission of the NVM 250. For scheduling, various schemes can be used, including First-In-First-Out (FIFO), Round Robin (RR), Deficit Round Robin (DRR), Weighted Round Robin (WRR), and Proportional Fair (PF).

[0045] MU 218 can monitor the status of buffer memory 216. For example, MU 218 can monitor buffer memory utilization, used buffer memory capacity, remaining buffer memory capacity, input and output data, latency for sending data after a host data request, data volume sent from NVM 250 to buffer memory 216, data volume sent from buffer memory 216 to host 100, data volume sent from host 100 to buffer memory 216, and data volume sent from buffer memory 216 to NVM 250. MU 218 can transmit the monitored data to CCU 217. For example, when buffer memory utilization exceeds a preset limit, MU 218 can transmit a control signal or trigger signal to CCU 217.

[0046] An error correction code (ECC) engine (not shown) performs error detection and correction on read data from the NVM 250. More specifically, the ECC engine generates parity bits for the data to be written to the NVM 250, and these generated parity bits are stored in the NVM 250 along with the write data. When reading data from the NVM 250, the ECC engine can correct errors in the read data by using the parity bits read from the NVM 250 along with the read data, and output the error-corrected read data.

[0047] The Advanced Encryption Standard (AES) engine (not shown) can perform at least one of encryption and decryption operations on the data input to the storage controller 210 by using a symmetric key algorithm.

[0048] The storage controller 210 may also include a bandwidth adjustment module (not shown). The bandwidth adjustment module includes hardware and software configured to perform bandwidth adjustments for adjusting the interface processing speed with the fabric network system. When an internal interface adjustment request is received from the fabric network system, the storage controller 210 can adjust the interface bandwidth of the storage device 200 based on the internal interface adjustment request using the bandwidth adjustment module. As the interface bandwidth of the storage device 200 increases, the interface processing speed increases, and the power consumption increases. Conversely, as the interface bandwidth of the storage device 200 decreases, the interface processing speed decreases, and the power consumption decreases.

[0049] Figure 3 This is a flowchart illustrating the operation of a storage device according to at least one exemplary embodiment of the present invention.

[0050] refer to Figure 1 and Figure 3During operation S110, storage device 200 can monitor the status of buffer memory 240 through storage controller 210. The monitored status of buffer memory 240 may include buffer memory utilization, used buffer memory capacity, remaining buffer memory capacity, data volume sent from NVM 250 to buffer memory 240, data volume sent from buffer memory 240 to host 100, data volume sent from host 100 to buffer memory 240, and data volume sent from buffer memory 240 to NVM 250, etc.

[0051] In operation S120, the storage controller 210 can monitor the status of the buffer memory 240 and determine whether the utilization rate of the buffer memory 240 is greater than a preset reference value. In operation S130, when the utilization rate of the buffer memory 240 is at or below the preset reference value, the storage controller 210 may operate in normal mode (as per the reference value) instead of congestion control mode. Figure 4 (Detailed description)

[0052] In operation S140, when the utilization rate of the buffer memory 240 exceeds a preset reference value, the storage controller 210 can operate in congestion control mode. For example, in congestion control mode, the storage controller 210 can generate data tokens corresponding to the remaining buffer memory capacity, and can allocate the generated data tokens to each of the multiple NVM devices 251, 252, 253, ... in the NVM 250 according to a preset algorithm. Each of the multiple NVM devices 251, 252, 253, ... in the NVM 250 can only send data corresponding to the allocated data token; therefore, the storage controller 210 can manage the utilization rate of the buffer memory 240 by using data tokens.

[0053] The storage controller 210 can monitor the status of the buffer memory 240 while operating in congestion control mode, and when the utilization of the buffer memory 240 is at or below a preset reference value, the storage controller 210 can resume operation in normal mode.

[0054] Figure 4 This is a signaling diagram of the operation of a storage device according to at least one exemplary embodiment of the present invention.

[0055] refer to Figure 1 and Figure 4In normal mode, operating the storage controller 210 may include the following operations: monitoring the status of the buffer memory 240; and controlling data transfer. The status monitoring operation may include operations S210, S212, and S214. In operation S210, the storage controller 210 may send a status request command to the buffer memory 240. In operation S212, the storage controller 210 may receive a requested status response from the buffer memory 240. The status response may include buffer memory utilization, used buffer memory capacity, remaining buffer memory capacity, data volume sent from the NVM 250 to the buffer memory 240, data volume sent from the buffer memory 240 to the host 100, data volume sent from the host 100 to the buffer memory 240, and data volume sent from the buffer memory 240 to the NVM 250, etc.

[0056] In operation S214, the storage controller 210 can determine the operating mode (normal mode or congestion control mode) based on the received status response and the buffer memory utilization.

[0057] Status monitoring operations can be performed in the middle of data transmission control operations (such as status update operations S226 and S232). Although Figure 4 Only the state update operations S226 and S232 are shown, but the storage controller 210 may additionally perform operations such as requesting the buffer memory 240 to update its state and determining the operating mode (normal mode or congestion control mode) after the state update.

[0058] Host 100 can request data from storage device 200. In operation S220, storage controller 210 can receive data requests from host 100. In operation S222, in normal mode, storage controller 210 can transmit the data request to NVM 250. In operation S224, NVM 250 can send data corresponding to the data request to buffer memory 240. Buffer memory 240 can store the received data until the received data is sent to host 100. In operation S230, storage controller 210 can control buffer memory 240 to send the stored data to host 100. Storage device 200 can receive data requests from one or more hosts 100 and control data transmission through storage controller 210.

[0059] The method of operating the storage controller 210 may also include the following operations: identifying the host 100 or the storage device 250 for data transfer between the host 100 or the storage device 250; establishing a connection including a 3-way handshake, a 4-way handshake, etc.; and handling errors that occur during data transfer.

[0060] Figure 5 This is a signaling diagram of a storage device in congestion control mode according to at least one example embodiment of the present invention.

[0061] refer to Figure 1 and Figure 5 The method of operating the storage controller 210 in congestion control mode may include the following operations: monitoring the status of the buffer memory 240; controlling data transfer; and managing data tokens. The status monitoring operation may include operations S310, S312, and S314. In operation S310, the storage controller 210 may send a status request command to the buffer memory 240. In operation S312, the storage controller 210 may receive a requested status response from the buffer memory 240. The status response may include buffer memory utilization, used buffer memory capacity, remaining buffer memory capacity, data volume sent from the NVM 250 to the buffer memory 240, data volume sent from the buffer memory 240 to the host 100, data volume sent from the host 100 to the buffer memory 240, and data volume sent from the buffer memory 240 to the NVM 250, etc.

[0062] In operation S314, the storage controller 210 can determine the operating mode (normal mode or congestion control mode) based on the received status response and the buffer memory utilization. For example, when the utilization of the buffer memory 240 is greater than a preset reference value, the storage controller 210 can change the operating mode from normal mode to congestion control mode.

[0063] Status monitoring operations can be performed in the middle of data transmission control operations (such as status update operations S336 and S342). Although Figure 5 Only state update operations S336 and S342 are shown, but the storage controller 210 may additionally perform the following operations: request the buffer memory 240 to update its state; and check the buffer memory utilization and determine the operating mode (normal mode or congestion control mode) after the state update.

[0064] In the data token management operation, the storage controller 210 can generate a data token in operation S320, distribute the data token to each of the multiple NVM devices 251, 252, 253, ... in the NVM 250 according to a preset algorithm in operation S322, control the data transmission of the NVM 250 based on the distributed data tokens in operation S334, and monitor the data token holding or utilization of the NVM 250.

[0065] In congestion control mode, storage controller 210 can generate data tokens based on buffer memory utilization or remaining buffer memory capacity in operation S320, and allocate the generated data tokens to NVM 250 according to a preset algorithm in operation S322. Furthermore, when the remaining buffer memory capacity increases by sending data from buffer memory 240 to host 100, storage controller 210 can generate additional data tokens or change the operating mode back to normal mode.

[0066] Host 100 can request data from storage device 200. In operation S330, storage controller 210 can receive data requests from host 100.

[0067] In operation S332, storage controller 210 can transmit a data request to NVM 250. In operation S334, NVM 250 can send data to buffer memory 240 using a data token corresponding to the requested data amount. When there are insufficient data tokens for NVM 250, storage controller 210 can control buffer memory 240 not to send data, regardless of the data request. For example, when NVM 250 holds one data token, it can be allowed to send data of size c, and when NVM 250 holds n data tokens, it can be allowed to send data of size n*c. When data of a specific size is sent from NVM 250 to buffer memory 240, the data token corresponding to that specific size of data may be consumed.

[0068] The buffer memory 240 can store received data until the received data is sent to the host 100. In operation S340, the storage controller 210 can control the buffer memory 240 to send the stored data to the host 100. When the data stored in the buffer memory 240 is sent to the host 100, the remaining buffer memory capacity increases, so the storage controller 210 can generate additional data tokens.

[0069] In the foregoing embodiments, the storage controller 210 is described operating in normal mode or congestion control mode based on reference values; however, the storage controller 210 is not limited thereto. According to various embodiments, the storage controller 210 may operate in multiple modes based on at least two thresholds. For example, the congestion control mode can be divided into multiple stages based on buffer memory utilization. For instance, when buffer memory utilization is at or below a first limit value, the storage controller 210 may operate in normal mode; when buffer memory utilization is greater than the first limit value and less than or equal to a second limit value, the storage controller 210 may operate in a first stage of controlling data transfer volume using data tokens; and when buffer memory utilization is greater than the second limit value, the storage controller 210 may operate in a second stage of temporarily suspending data transfer. By further subdividing the scheduling method and data transfer volume control method, the congestion control mode may include more stages.

[0070] In addition, the storage controller 210 can estimate the buffer memory utilization by monitoring the buffer memory input and output data, data transfer rate, latency, etc., and operate in congestion control mode based on the estimated buffer memory utilization.

[0071] In the foregoing embodiments, the storage controller 210 is described as issuing data tokens and controlling data transmission volume, bandwidth, etc. in congestion control mode. However, the data transmission volume control method is not limited to this, and the NVM 250 can control data transmission volume, bandwidth, etc. based on the received data tokens.

[0072] Figure 6 This is a flowchart of an operation for managing data tokens in a storage device, based on at least one example embodiment of the present invention.

[0073] refer to Figure 1 and Figure 6 In operation S410, storage controller 210 can generate data tokens in congestion control mode. Storage controller 210 can generate data tokens based on buffer memory utilization or remaining buffer memory capacity, and can generate additional data tokens when the remaining buffer memory capacity is increased by sending data stored in buffer memory 240 to host 100.

[0074] In operation S420, the storage controller 210 can allocate data tokens to each of the multiple NVM devices 251, 252, 253, ... in the NVM 250 according to a preset algorithm. The storage controller 210 can allocate data tokens according to token allocation rules, which include the data request order of the host 100, the requested data size, the waiting time after the data request, the request priority set by the host 100, and the priority for data or NVM 250 classification.

[0075] In operation S430, storage controller 210 can control data transfer on NVM 250 based on the assigned data token. In response to a data request from host 100, storage controller 210 can control data transfer by considering the data token assigned to NVM 250. When data is sent to buffer memory 240, storage controller 210 can retrieve the corresponding data token.

[0076] Figure 7 It is a table of congestion control signals (CCS) of a storage device according to at least one example embodiment of the present invention.

[0077] refer to Figure 1 and Figure 7 The storage controller 210 can control the NVM 250 by sending CCS signals via bus 220. In addition to Ethernet, PCIe, SCSI, SATA, or IDE interfaces for data communication, bus 220 may also include interfaces for device control, such as an Inter-Integrated Circuit (I2C) interface, a Universal Asynchronous Serial Receiver-Transmitter (UART) interface, and a Serial Peripheral Interface (SPI). For example, the first interface circuitry for data communication can use an Ethernet interface, and the second interface circuitry for transmitting device control signals can use an I2C interface.

[0078] CCS may include a memory ID for identifying the NVM 250, a control mode indicating normal mode or congestion control mode, a data token including data token information assigned to the NVM 250, and the priority and weight of each data request.

[0079] Figure 8 It is a priority table of a storage device according to at least one example embodiment of the present invention.

[0080] For example, the priority and weight of the first NVM NVM1 can be set to 1 and 4 respectively, the priority and weight of the second NVM NVM2 can be set to 2 and 2 respectively, the priority and weight of the third NVM NVM3 can be set to 3 and 2 respectively, and the priority and weight of the fourth NVM NVM4 can be set to 4 and 1 respectively.

[0081] In this embodiment, by considering weights, the storage controller 210 can allocate four data tokens to the first NVMNVM1, two data tokens to the second NVMNVM2, two data tokens to the third NVMNVM3, and one data token to the fourth NVMNVM4. When there are remaining data tokens, the storage controller 210 can allocate additional data tokens based on priority. For example, when there are seven data tokens in the second allocation, the storage controller 210 can first allocate four data tokens to the first NVMNVM1, then allocate two data tokens to the second NVMNVM2. The storage controller 210 can then allocate the remaining data token to the third NVMNVM3 and end the second allocation.

[0082] In another embodiment, the storage controller 210 may allocate all required data tokens to the first NVM NVM1 with priority 1, and allocate the remaining data tokens to NVMs with lower priorities. Various token allocation methods are possible, and the inventive concept is not limited to the allocation methods described above.

[0083] In another embodiment, the storage controller 210 may first allocate the necessary data tokens according to the priority of the NVM, and then allocate the data tokens to NVMs with the same or substantially the same priority according to the weight of the NVM.

[0084] Figure 9 This is a diagram illustrating a congestion control method for a storage device according to at least one exemplary embodiment of the present invention.

[0085] As scheduling schemes for congestion control of storage devices, FIFO, RR, DRR, WRR, PF, and other schemes can be used.

[0086] The first-come, first-served (FCFS) scheme is a very simple scheduling scheme that allocates resources to the NVM that requests data transfer first.

[0087] The RR (Redirect Resource) scheme is a method of allocating resources sequentially to multiple NVMs, and for example, using an algorithm that runs packets of the same or substantially the same priority sequentially at time slice intervals. WRR scheduling can assign weights to each of the NVMs to allocate different amounts of data processing to them.

[0088] In the DRR scheme, a series of NVMs are served using an RR (Redirect Ratio) scheduler. Unlike the WRR (Wait-Redirect Ratio) scheme, the DRR scheme is a scheduling scheme designed to take data length into account when serving. The maximum data size is subtracted from the current data size, and data larger than the maximum data size is paused until the nested scheduler arrives. According to at least some example embodiments, the value of the maximum data size can be determined based on the preferences of the designers and / or operators of storage device 200 and / or storage controller 210 (e.g., based on empirical analysis of potential values).

[0089] Weighted DRR (WDRR) is an improved DRR algorithm that adds an additional weighting function to the DRR scheduler. This not only guarantees the processing capacity of each class but also provides appropriate access for excessive bandwidth. In WDRR, different weights can be assigned to NVMs to guarantee the Quality of Service (QoS) requirements requested by several service flows.

[0090] PF is a resource allocation scheme based on the rank of the ratio of the currently available rate to the average rate of each traffic flow. The scheme includes a first phase where the scheduler discovers a traffic flow with the best ratio of the currently available rate to the average rate and allocates resources to the discovered traffic flow; and a second phase where the average rate value of all traffic flows is updated even when the traffic flow has no packets to send.

[0091] For example, refer to Figure 9 Congestion control methods can be combined and multiple scheduling methods can be used in combination. The priority of the first NVM NVM1 can be classified as very urgent, the priority of the second NVM NVM2 and the third NVM NVM3 can be classified as urgent, the priority of the fourth to sixth NVM NVM4, NVM5 and NVM6 can be classified as high, the priority of the seventh to ninth NVM NVM7, NVM8 and NVM9 can be classified as medium, and the priority of the tenth to twelfth NVM NVM10, NVM11 and NVM12 can be classified as low.

[0092] Data from the first NVM (NVM1) can be classified as first priority and processed first. Once the data from the first NVM (NVM1) has been completely transmitted, data from the second NVM (NVM2) and the third NVM (NVM3) can be processed. Because the second NVM (NVM2) and the third NVM (NVM3) have the same or nearly the same priority, they can transmit data using a RR (Redirect-Redirect) scheme.

[0093] Resources can be allocated to NVM groups categorized based on high, medium, and low priorities using a WRR scheme. Higher weights can be assigned to groups with higher priorities, and lower weights can be assigned to groups with lower priorities. Resources can be allocated to NVMs within each group using a RR scheme (e.g., the fourth through sixth NVMs NVM4, NVM5, and NVM6 in the high-priority group).

[0094] Furthermore, the congestion control method can also limit the data transfer volume of the NVM by using data tokens. For example, even when resources are allocated according to the scheduling scheme described above, the data transfer volume of each NVM can be limited based on the number of data tokens received from the storage controller 210. The NVM can send a data volume corresponding to the number of data tokens received, and when data tokens are consumed, data transfer can be limited until additional data tokens are received.

[0095] Figure 10 This is a block diagram of a data center 3000 that applies a storage device according to at least one example embodiment of the concept of the present invention.

[0096] refer to Figure 10 Data center 3000 is a facility used to collect various types of data and provide services, and may be referred to as a data storage center. Data center 3000 may be a system used to operate search engines and databases, or a computing system used in enterprises such as banks or government agencies. Data center 3000 may include application servers 3100 to 3100n and storage servers 3200 to 3200m. According to embodiments, the number of application servers 3100 to 3100n and the number of storage servers 3200 to 3200m may be selected differently, and the number of application servers 3100 to 3100n may differ from the number of storage servers 3200 to 3200m.

[0097] Application server 3100 or storage server 3200 may include at least one of processor 3110 or 3210 and memory 3120 or 3220. When storage server 3200 is described as an example, processor 3210 may control the general operation of storage server 3200 and access memory 3220 to execute instructions and / or data loaded on memory 3220. Memory 3220 may include dual data rate synchronous DRAM (DDR SDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), dual in-line memory module (DIMM), Optane DIMM, or non-volatile DIMM (NVDIMM). According to embodiments, the number of processors 3210 and memory 3220 included in storage server 3200 may be selected differently. In embodiments, processors 3210 and memory 3220 may provide processor-memory pairs. In embodiments, the number of processors 3210 may differ from the number of memory 3220. Processor 3210 may include a single-core processor or a multi-core processor. The description of storage server 3200 can be similarly applied to application server 3100. According to an embodiment, application server 3100 may not include storage device 3150. Storage server 3200 may include at least one storage device 3250. According to an embodiment, the number of storage devices 3250 included in storage server 3200 may be selected differently.

[0098] Application servers 3100 to 3100n and storage servers 3200 to 3200m can communicate with each other via network 3300. Network 3300 can be implemented using FC, Ethernet, etc. Here, FC is a medium used for relatively high-speed data transmission and can be used with an optical switch configured to provide high performance / high availability. Depending on the access method of network 3300, storage servers 3200 to 3200m can be provided as file storage, block storage, or object storage.

[0099] In this embodiment, network 3300 can be a storage-specific network, such as a storage area network (SAN). For example, the SAN can be an FC-SAN implemented using an FC network and according to the FC protocol (FCP). As another example, the SAN can be an IP-SAN implemented using a Transmission Control Protocol (TCP) / Internet Protocol (IP) network and according to a TCP / IP-based SCSI or Internet SCSI (iSCSI) protocol. In another embodiment, network 3300 can be a general network, such as a TCP / IP network. For example, network 3300 can be implemented according to an Ethernet-based FC (FCoE) protocol, a Network Attached Storage (NAS) protocol, an architecture-based NVMe (NVMe over Fabrics, NVMe-oF) protocol, etc.

[0100] The following text will primarily describe application server 3100 and storage server 3200. The description of application server 3100 can be applied to another application server 3100n, and the description of storage server 3200 can be applied to another storage server 3200m.

[0101] Application server 3100 can store data requested by users or clients in one of storage servers 3200 to 3200m via network 3300. Furthermore, application server 3100 can retrieve data requested by users or clients from one of storage servers 3200 to 3200m via network 3300. For example, application server 3100 can be implemented as a web server, database management system (DBMS), etc.

[0102] Application server 3100 can access memory 3120n or storage device 3150n included in another application server 3100n via network 3300, or access memory 3220 to 3220m or storage device 3250 to 3250m included in storage servers 3200 to 3200m via network 3300. In doing so, application server 3100 can perform various operations on data stored in application servers 3100 to 3100n and / or storage servers 3200 to 3200m. For example, application server 3100 can execute instructions for moving or copying data between application servers 3100 to 3100n and / or storage servers 3200 to 3200m. According to at least some exemplary embodiments of the present invention, data can be moved directly or via storage devices 3250 to 3250m in storage servers 3200 to 3200m to storage devices 3220 to 3220m in application servers 3100 to 3100n to storage devices 3120 to 3120n. For security or privacy, data moved via network 3300 can be encrypted.

[0103] When describing storage server 3200 as an example, interface 3254 can provide physical connectivity between processor 3210 and controller 3251, as well as physical connectivity between network interface controller (NIC) 3240 and controller 3251. For example, interface 3254 can be implemented using a Direct Attached Storage (DAS) scheme that directly accesses storage device 3250 via a dedicated cable. Furthermore, interface 3254 can be implemented using various interface schemes, such as Advanced Technology Attachment (ATA) interface, Serial ATA (SATA) interface, External SATA (e-SATA) interface, SCSI, Serial Attached SCSI (SAS), Peripheral Component Interconnect (PCI) interface, PCIe interface, NVM High Speed ​​(NVMe) interface, IEEE 1394 interface, Universal Serial Bus (USB) interface, Secure Digital (SD) card interface, Multimedia Card (MMC) interface, Embedded Multimedia Card (eMMC) interface, UFS interface, Embedded Universal Flash (eUFS), or Compact Flash (CF) interface.

[0104] Storage server 3200 may also include switch 3230 and NIC 3240. Switch 3230 can selectively connect processor 3210 or NIC 3240 to storage device 3250 under the control of processor 3210.

[0105] In this embodiment, NIC 3240 may include a network interface card, a network adapter, etc. NIC 3240 can connect to network 3300 via a wired interface, wireless interface, Bluetooth interface, optical interface, etc. NIC 3240 may include internal memory, a digital signal processor (DSP), a host bus interface, etc., and can connect to processor 3210, switch 3230, etc., via the host bus interface. The host bus interface can be implemented using one of the examples of interface 3254 described above. In this embodiment, NIC 3240 can be integrated with at least one of processor 3210, switch 3230, and storage device 3250.

[0106] Processors 3110 to 3110n in application servers 3100 to 3100n or processors 3210 to 3210m in storage servers 3200 to 3200m can program or read data by sending commands to storage devices 3150 to 3150n or 3250 to 3250m or memory 3120 to 3120n or 3220 to 3220m. According to at least some exemplary embodiments of the present invention, data can be error-corrected using an ECC engine. The data is processed by Data Bus Inversion (DBI) or Data Masking (DM) and may include Cyclic Redundancy Check (CRC) information. For security or privacy, the data may be encrypted.

[0107] Storage devices 3150 to 3150n or 3250 to 3250m can send control signals and command / address signals to NAND flash memory devices 3252 to 3252m in response to commands received from processors 3110 to 3110n or 3210 to 3210m. Therefore, when reading data from NAND flash memory devices 3252 to 3252m, a read enable (RE) signal can be input as a data output control signal to output data via the digital output (DQ) bus. A data strobe (DQS) can be generated using the RE signal. Command / address signals can be latched to the page buffer at the leading or trailing edge of the write enable (WE) signal.

[0108] Controller 3251 can control the general operation of storage device 3250. In embodiments, controller 3251 may include SRAM. Controller 3251 can write data to NAND flash memory device 3252 in response to a write command, or read data from NAND flash memory device 3252 in response to a read command. For example, write and / or read commands may be provided from processor 3210 in storage server 3200, processor 3210m in another storage server 3200m, or processor 3110 or 3110n in application server 3100 or 3100n. DRAM 3253 may temporarily store (buffer) data to be written to or read from NAND flash memory device 3252. In addition, DRAM 3253 may store metadata. Here, metadata is generated by controller 3251 to manage user data or data in NAND flash memory device 3252. For security or privacy, storage device 3250 may include a secure element (SE).

[0109] Exemplary embodiments of the inventive concept have been described, and it will be apparent that they can be varied in many ways. Such variations should not be considered as departing from the intended spirit and scope of the exemplary embodiments of the inventive concept, and it will be clear to those skilled in the art that all such modifications are intended to be included within the scope of the appended claims.

Claims

1. A storage device, comprising: Buffer memory, configured to temporarily store data; Multiple non-volatile memory devices; The storage controller circuit is configured as follows: Buffer state information is generated by monitoring the state of the buffer memory. By setting buffer data transfer authorization for non-volatile memory devices based on the generated buffer state information, operation is performed in congestion control mode. Data tokens are generated based on buffer memory utilization. Data tokens are assigned to multiple non-volatile memory devices based on a priority table, which indicates the data transfer priority of the multiple non-volatile memory devices; as well as Data transfer between multiple non-volatile memory devices is controlled based on the assigned data token; as well as The first interface circuit is configured to communicate with the memory controller circuit and multiple non-volatile memory devices. The first interface circuit is connected to the network via an Ethernet interface.

2. The storage device according to claim 1, wherein, The plurality of non-volatile memory devices are configured to perform data transfer rate control based on information received from the memory controller circuitry, including buffer memory data transfer authorization.

3. The storage device according to claim 1, in, The memory controller circuit is configured to control data transfer of multiple non-volatile memory devices based on a priority table.

4. The storage device according to claim 3, wherein, The priority table also includes a weight for each non-volatile memory device, which is set based on the data request order and the amount of data requested.

5. The storage device according to claim 1, wherein, The storage controller circuit is configured to control the data transfer of multiple non-volatile memory devices based on a priority table and data tokens according to a preset scheduling scheme.

6. The storage device according to claim 1, wherein, The storage controller circuit is configured such that the storage controller circuit has When the utilization of the buffer memory exceeds the first limit, the first phase of operation in congestion control mode, and When the utilization rate of the buffer memory exceeds the second limit, the second phase of data transmission is temporarily suspended.

7. The storage device according to claim 1, wherein, The storage controller circuit is configured to control at least one of the bandwidth, data transfer rate, or power consumption of the first interface circuit based on the utilization of the buffer memory.

8. The storage device according to claim 1, further comprising: The second interface circuit is configured to control the amount of data transfer between multiple non-volatile memory devices via the memory controller circuit.

9. The storage device according to claim 1, wherein, The storage controller circuit is configured to, Buffer memory utilization is estimated by monitoring the input and output data and data transfer rate of the buffer memory. Based on the estimated buffer memory utilization, operation is performed in congestion control mode.

10. A method of operating a storage device, the method comprising: Buffer state information is generated by monitoring the state of the buffer memory; The data transfer mode of multiple non-volatile memory devices is set based on the generated buffer memory state information; Control multiple non-volatile memory devices based on the set data transfer mode; Data tokens are generated based on the utilization of the buffer memory; Data tokens are assigned to multiple non-volatile memory devices based on a priority table, which indicates the data transfer priority of the multiple non-volatile memory devices; as well as Data transfer between multiple non-volatile memory devices is controlled based on the assigned data token. The data transmission mode includes a congestion control mode, which includes setting data transmission authorization for non-volatile memory devices in response to the utilization of the buffer memory being greater than a preset reference value.

11. The method according to claim 10, in, The control includes controlling data transfers across multiple non-volatile memory devices based on a priority table.

12. The method according to claim 11, wherein, The priority table includes a weight for each non-volatile memory device, which is set based on the data request order and the amount of data requested.

13. The method according to claim 12, wherein, The control also includes: Data transfer between multiple non-volatile memory devices is controlled based on a priority table and data tokens according to a preset scheduling scheme.

14. The method of claim 10, further comprising: By using a first interface circuit to communicate with the memory controller circuit and multiple non-volatile memory devices, The first interface circuit is connected to the network via an Ethernet interface.

15. The method according to claim 10, wherein, The control includes: The bandwidth, data transmission rate, or power consumption of the first interface circuit is controlled based on the utilization of the buffer memory.

16. The method of claim 10, further comprising: The data transfer volume of multiple non-volatile memory devices is controlled through a second interface circuit.

17. The method according to claim 10, wherein, The control includes: The utilization rate of the buffer memory is estimated by monitoring the input and output data and data transfer rate of the buffer memory; and Based on the estimated buffer memory utilization, operation is performed in congestion control mode.

18. A non-transitory computer-readable recording medium having computer-readable instructions recorded thereon, which, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: Buffer state information is generated by monitoring the state of the buffer memory; The data transfer mode of multiple non-volatile memory devices is set based on the generated buffer memory state information; Control multiple non-volatile memory devices based on the set data transfer mode; Data tokens are generated based on the utilization of the buffer memory; Data tokens are assigned to multiple non-volatile memory devices based on a priority table, which indicates the data transfer priority of the multiple non-volatile memory devices; as well as Data transfer between multiple non-volatile memory devices is controlled based on the assigned data token. The data transmission mode includes a congestion control mode, which includes setting data transmission authorization for non-volatile memory devices in response to the utilization of the buffer memory being greater than a preset reference value.

Images