Storage device, storage system, and method for operating the storage device and storage system

Abstract

A storage device, a method for operating the same, and a storage system. The storage device includes: a first memory; a second memory configured to store a bitmap indicating whether the data stored in the first memory is dirty data; and a storage controller configured to control the first memory and the second memory, the storage controller configured to: receive a transfer command from a CXL (Computing Fast Link) switch to transmit dirty data stored in the first memory to a first external device connected to the CXL switch; and in response to the transfer command, prefetching the data stored in the first memory into the second memory based on the bitmap, and sending a request command to the CXL switch to request cache refresh from at least one second external device connected with the CXL switch.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10 - 2024 - 0010845, filed with the Korean Intellectual Property Office on January 24, 2024, the entire contents of which are incorporated herein by reference. Technical field

[0003] The present inventive concept relates to a storage device, a storage system, and methods of operating the storage device and the storage system. Background art

[0004] With the development of technologies such as artificial intelligence (AI), big data, and edge computing, there is a growing need to process large amounts of data faster on devices. In other words, it can be advantageous to provide high - bandwidth applications that perform complex computations with faster data processing and more efficient memory access.

[0005] To meet this need, data processing using a CXL (Compute Express Link) interface is being utilized. A host - managed device memory (HDM) is provided in / located in a CXL device. The HDM is a memory area existing inside the CXL device and can be accessed by an HA (e.g., a host CPU) or another CXL device at its physical address. It may be necessary to manage dirty data, which is newly updated data among the data stored in the HDM, to maintain its consistency in data transfer between devices. Therefore, research is being conducted on this. Summary of the invention

[0006] Some example embodiments aim to achieve a technical objective of providing a storage device with improved reliability and data transfer speed, a method for operating the storage device, and a storage system.

[0007] However, the present inventive concept is not limited thereto. Other objects and advantages not mentioned according to the present inventive concept can be understood based on the following description and can be more clearly understood based on some example embodiments of the present inventive concept. In addition, it will be readily understood that the objects and advantages according to the present inventive concept can be achieved using the devices and their combinations shown in the claims.

[0008] According to some example embodiments, a storage device is provided, including: a first memory; a second memory configured to store a bitmap indicating whether data stored in the first memory is dirty data; and a storage controller configured to control the first memory and the second memory. The storage controller is configured to: receive a transfer command from a CXL (Compute Express Link) switch to send dirty data stored in the first memory to a first external device connected to the CXL switch; and in response to the transfer command, prefetch the dirty data stored in the first memory into the second memory based on the bitmap; and send a request command to the CXL switch to request cache flushing from at least one second external device connected to the CXL switch.

[0009] According to some example embodiments, a method for operating a storage device is provided, the storage device including: a first memory including a first HDM (Host-Managed Device Memory) area; a second memory including: a second HDM area configured to store a bitmap configured to store bit values corresponding to data stored in the first memory; and a prefetch data area configured to store data prefetched from the first memory; and a storage controller configured to control the first memory and the second memory, the method including: receiving, by the storage controller, a transfer command from a CXL (Compute Express Link) switch, the transfer command being configured to indicate sending data corresponding to a first bit value of the bitmap among data stored in the first memory to a first external device connected to the CXL switch; in response to the transfer command, prefetching, by the storage controller, the data corresponding to the first bit value of the bitmap from the first memory into the second memory; and in response to the transfer command, sending, by the storage controller, a request command to the CXL switch, the request command being configured to request cache flushing of data corresponding to the first bit value and a second bit value of the bitmap from at least one second external device connected to the CXL switch.

[0010] According to some example embodiments, a storage system is provided, including: a host device; a first CXL (Compute Express Link) device; a second CXL device; and a CXL switch configured to connect the host device to the first CXL device and the second CXL device to each other via a CXL interface. The first CXL device includes: a first memory; a second memory configured to store a bitmap indicating whether data stored in the first memory is dirty data; and a storage controller configured to control the first memory and the second memory. The storage controller is configured to: receive a transfer command from the CXL switch to send data indicated as dirty data by the bitmap among the data stored in the first memory to the second CXL device; and in response to the transfer command, prefetch the data indicated as dirty data stored in the first memory to the second memory based on the bitmap, and send a request command to the CXL switch, the request command requesting a cache flush of data related to the transfer command. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other aspects and features of the inventive concept will become more apparent by referring to some illustrative example embodiments of the inventive concept described in detail with reference to the accompanying drawings, in which:

[0012] Figure 1 is an example diagram showing a computing system including a storage system according to some example embodiments;

[0013] Figure 2 is an example diagram showing Figure 1 components of a host and a CXL storage device in

[0014] Figure 3 is an example diagram showing Figure 2 the configuration of a dirty data transfer manager in

[0015] Figure 4 is an example diagram showing Figure 2 the configuration of a buffer memory in

[0016] Figure 5 is an example diagram showing Figure 4 the relationship between a bitmap and user data stored in a non-volatile memory in

[0017] Figure 6 is an example diagram showing Figure 2 an example block diagram of a non-volatile memory in

[0018] Figure 7 is a diagram showing a 3D V-NAND structure that can be applied to a non-volatile memory according to some example embodiments;

[0019] Figure 8 is a flowchart showing the operation of a storage system according to some example embodiments;

[0020] Figures 9 to 14 is showing according to some example embodiments Figure 8 a diagram of the operations shown; and

[0021] Figure 15 is an example diagram of a data center including a computing system according to some example embodiments. DETAILED DESCRIPTION

[0022] Hereinafter, example embodiments according to the inventive concept will be described with reference to the accompanying drawings.

[0023] Figure 1 is an example diagram of a computing system including a storage system according to some example embodiments.

[0024] Referring Figure 1 , the computing system 100 may include a plurality of hosts 101, 102, and 103, a plurality of storage devices 111a, 111b, 112a, 112b, 113a, and 113b, and a plurality of CXL (Compute Express Link) storage devices 130, 140, and 150. The plurality of CXL storage devices 130, 140, and 150 may constitute a storage system 100S.

[0025] In some example embodiments, the computing system 100 may be included in a user device (such as a personal computer, laptop computer, server, media player, digital camera, etc.) or a vehicle device (such as a navigation device, black box, vehicle electronic device, etc.), but the example embodiments are not limited thereto.

[0026] In some example embodiments, the computing system 100 may be a mobile system, such as a portable communication terminal (mobile phone), smart phone, tablet computer, personal computer (PC), wearable device, healthcare device, or IOT (Internet of Things) device.

[0027] The hosts 101, 102, and 103 may control the overall operation of the computing system 100. In some example embodiments, each of the hosts 101, 102, and 103 may be one of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a data processing unit (DPU), etc. In some example embodiments, each of the hosts 101, 102, and 103 may include a single-core processor or a multi-core processor. In addition, in some example embodiments, each of the hosts 101, 102, and 103 may include an accelerator.

[0028] Multiple storage devices 111a, 111b, 112a, 112b, 113a, and 113b can be used as the main memory or system memory of the computing system 100. The storage devices 111a and 111b can be connected to the host 101. The storage devices 112a and 112b can be connected to the host 102. The storage devices 113a and 113b can be connected to the host 103.

[0029] In some example embodiments, each of the multiple storage devices 111a, 111b, 112a, 112b, 113a, and 113b can be a dynamic random access memory (DRAM) device and can have a dual in-line memory module (DIMM) form factor. However, example embodiments of the inventive concept are not limited thereto, and each of the multiple storage devices 111a, 111b, 112a, 112b, 113a, and 113b can include non-volatile memory such as flash memory, phase change RAM (PRAM), resistive RAM (RRAM), and magnetic RAM (MRAM).

[0030] The storage devices 111a and 111b can communicate directly with the host 101 via a DDR interface (double data rate interface). The storage devices 112a and 112b can communicate directly with the host 102 via a DDR interface. The storage devices 113a and 113b can communicate directly with the host 103 via a DDR interface.

[0031] In some example embodiments, each of the hosts 101, 102, and 103 can include a storage controller configured to control the multiple storage devices 111a, 111b, 112a, 112b, 113a, and 113b, respectively. However, example embodiments of the inventive concept are not limited thereto, and the multiple storage devices 111a, 111b, 112a, 112b, 113a, and 113b can communicate with the hosts 101, 102, and 103 via various interfaces, respectively.

[0032] Multiple CXL storage devices 130, 140, and 150 can include CXL storage controllers 131, 141, and 151 and non-volatile memories 132, 142, and 152, respectively. The CXL storage controllers 131, 141, and 151 can store data in the non-volatile memories 132, 142, and 152, respectively, or can transmit or send the data stored in the non-volatile memories 132, 142, and 152 to the hosts 101, 102, and 103 under the control of the hosts 101, 102, and 103, respectively. In some example embodiments, each of the non-volatile memories 132, 142, and 152 can be a NAND flash memory. However, example embodiments of the inventive concept are not limited thereto.

[0033] In some example embodiments, hosts 101, 102, and 103 and CXL storage devices 130, 140, and 150 may be configured to share the same interface with each other. For example, hosts 101, 102, and 103 and CXL storage devices 130, 140, and 150 may communicate with each other via a CXL interface (Compute Express Link interface) 120. The CXL interface 120 may represent a low-latency and high-bandwidth link that is configured to support dynamic protocol multiplexing of coherence, memory access, and input / output protocols (IO protocols) to enable various connections between accelerators, storage devices, or various electronic devices.

[0034] Figure 2 is an example diagram showing Figure 1 each component in the host and CXL storage device in Figure 3 is an example diagram showing Figure 2 the configuration of the dirty data transfer manager in Figure 4 is an example diagram showing Figure 2 the configuration of the buffer memory (BFM) in Figure 5 is an example diagram showing Figure 4 the relationship between the bitmap and the user data stored in the non-volatile memory in

[0035] Figure 2 Only hosts 101 and CXL storage device 130 are shown by way of example. The following description of host 101 may equally apply to Figure 1 each of hosts 102 and 103 in Figure 1 each of CXL storage devices 140 and 150 in

[0036] Host 101 and CXL storage device 130 may communicate with each other via CXL interface 120. However, example embodiments of the inventive concept are not limited thereto, and host 101 and CXL storage device 130 may communicate with each other based on various computing interfaces such as the GEN-Z protocol, the NVLink protocol, the CCIX (Cache Coherence Interconnect for Accelerators) protocol, or the OpenCAPI (Coherent Accelerator Processor Interface) protocol.

[0037] See Figure 2, the CXL interface 120 may include low-level protocols such as CXL.io, CXL.cache, CXL.mem, etc. The CXL.io protocol may be a PCIe (Peripheral Component Interconnect Express) transaction layer and may be used to search for devices, manage interrupts, provide or allow access to registers, handle initialization, and handle signal errors in the computing system 100. When an accelerator (e.g., a GPU or a Field Programmable Gate Array (FPGA)) accesses the host memory 101c, the CXL.cache protocol may be used. When the host 101 accesses the dedicated memory of the accelerator or the buffer memory 133 of the CXL storage device 130, the CXL.mem protocol may be used.

[0038] In some example embodiments, the host 101 and the CXL storage device 130 may communicate with each other using the input / output protocol CXL.io. CXL.io may have a PCIe-based inconsistent input / output protocol. The host 101 and the CXL storage device 130 may use CXL.io to exchange various information including user data (UD) 132a with each other.

[0039] The host 101 may include a host processor 101b, a host memory 101c, and a CXL host interface (CXL_H I / F) circuit 101a. The host processor 101b may control all operations of the host 101. In some example embodiments, the host processor 101b may be one of multiple modules provided in an application processor (AP). The application processor may be embodied as a System on Chip (SoC).

[0040] The host memory 101c may be a working memory and may store instructions, programs, and / or data required for the operation of the host processor 101b therein. In some example embodiments, the host memory 101c may act as a buffer memory for temporarily storing data to be transferred or sent to the CXL storage device 130, or data transferred or sent from the CXL storage device 130. When the host processor 101b is implemented as an AP, the host memory 101c may be an embedded memory provided within the AP, or may be a non-volatile memory or a memory module provided outside the AP.

[0041] In some example embodiments, the host processor 101b and the host memory 101c may be implemented as separate semiconductor chips. Alternatively, in some example embodiments, the host processor 101b and the host memory 101c may be integrated into a single semiconductor chip.

[0042] The CXL host interface circuit 101a can communicate with the CXL storage device 130 via the CXL interface 120. For example, the CXL host interface circuit 101a can communicate with the CXL storage device 130 via the CXL switch 120a included in the CXL interface 120.

[0043] For example, the CXL switch 120a can perform management of data storage, data update, and data transfer between devices (such as host devices, storage devices, accelerators, etc.) connected to the CXL interface 120. In some example embodiments, the CXL switch 120a can have a hardware configuration for implementing this function. However, the example embodiments of the inventive concept are not limited thereto.

[0044] The CXL storage device 130 can include a CXL storage controller 131, a buffer memory 133, and a non-volatile memory 132.

[0045] The CXL storage controller 131 can include a CXL storage interface (CXL_S I / F) circuit 131a, a processor 131b, an internal buffer 131c, a dirty data transfer manager (DDTM) 131d, an error correction code (ECC) engine 131e, and a buffer memory interface (I / F) circuit 131f. According to some example embodiments, each of the CXL storage controller 131 and the CXL storage interface circuit 131a, processor 131b, internal buffer 131c, dirty data transfer manager (DDTM) 131d, error correction code engine 131e, and buffer memory interface circuit 131f can include or be implemented as: one or more processing circuits (such as hardware including logic circuits); a hardware / software combination such as a processor 131b that executes software; or a combination thereof. For example, the processing circuit can more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc.

[0046] The CXL storage interface circuit 131a can be connected to the CXL switch 120a. The CXL storage interface circuit 131a can communicate with the host 101 or other CXL storage devices (such as Figure 1 the CXL storage devices 140 and 150 shown) via the CXL switch 120a.

[0047] The processor 131b can be configured to control the overall operation of the CXL storage controller 131. Firmware for controlling the operation of the CXL storage controller 131 can be executed by the processor 131b.

[0048] The internal buffer 131c can be used as a working memory or an internal buffer memory of the CXL storage controller 131. In some example embodiments, the internal buffer 131c can receive data stored in the prefetch data area 133c of the buffer memory 133 and store the data. In some example embodiments, the internal buffer 131c can store therein data refreshed from the cache memory of a device (e.g., a host device, an accelerator, etc.) connected to the CXL interface 120. In some example embodiments, the internal buffer 131c can be used to update data provided from the prefetch data area 133c of the buffer memory 133 based on data refreshed from the cache memory of a device (e.g., a host device, an accelerator, etc.) connected to the CXL interface 120.

[0049] In some example embodiments, the CXL storage controller 131 can transmit data refreshed from the cache memory of a device (e.g., a host device, an accelerator, etc.) connected to the CXL interface 120 or data updated based on the refreshed data to other CXL storage devices (e.g., Figure 1 the CXL storage devices 140 and 150 shown). A detailed description related thereto according to some example embodiments will be made later.

[0050] In some example embodiments, the internal buffer 131c can be implemented as, for example, SRAM (static random access memory). However, the example embodiments of the inventive concept are not limited thereto.

[0051] In some example embodiments, the internal buffer 131c may not be provided / located within the storage controller 131, but may be provided / located outside the storage controller 131.

[0052] The dirty data transfer manager (DDTM) 131d can manage the dirty data stored in the non-volatile memory 132 and can manage operations of transferring or sending the dirty data to other CXL storage devices (e.g., Figure 1 the CXL storage devices 140 and 150 shown).

[0053] According to some example embodiments, dirty data refers to newly updated data among the user data 132a stored in the non-volatile memory 132 according to write commands from a host device (e.g., Figure 1 the host devices 101, 102, and 103 shown) or other devices connected to the CXL interface 120.

[0054] Reference Figure 2 and Figure 3 , in some example embodiments, the dirty data transfer manager (DDTM) 131d can include a dirty tracker DT, a bitmap checker BC, a prefetch requester PFR, and a cache flush requester CFR.

[0055] In some example embodiments, at least some of the dirty tracker DT, the bitmap checker BC, the prefetch requester PFR, and the cache flush requester CFR may be implemented in software, while the rest may be implemented in hardware. However, the example embodiments are not limited thereto, and all of the dirty tracker DT, the bitmap checker BC, the prefetch requester PFR, and the cache flush requester CFR may be implemented in software or hardware. For example, the dirty data transfer manager (DDTM) 131d including each of the dirty tracker DT, the bitmap checker BC, the prefetch requester PFR, and the cache flush requester CFR may include or be implemented as: one or more processing circuits (such as hardware including logic circuits); a hardware / software combination such as a processor that executes software; or a combination thereof.

[0056] The dirty tracker DT may manage a bitmap (e.g., Figure 4 the BM shown) corresponding to a physical address indicated by a write command that is received from a host device (e.g., Figure 1 the host devices 101, 102, and 103 shown) or other devices connected to the CXL interface 120. For example, the dirty tracker DT may manage the bitmap (e.g., Figure 4 the BM shown) by setting the bit value 1 indicating dirty data to the position corresponding to the physical address indicated by the write command in the bitmap (e.g., Figure 4 the BM shown), where the write command is received from a host device (e.g., Figure 1 the host devices 101, 102, and 103 shown) or other devices connected to the CXL interface 120.

[0057] The bitmap checker BC may check the bitmap (e.g., Figure 4 the BM shown). For example, when the bit value of the position of the bitmap (e.g., Figure 4 the BM shown) is 1, the bitmap checker BC may identify or determine that the user data 132a of the non-volatile memory 132 corresponding thereto is dirty data. For example, when the bit value of the position of the bitmap (e.g., Figure 4 the BM shown) is 0, the bitmap checker BC may identify or determine that the user data 132a of the non-volatile memory 132 corresponding thereto is not dirty data.

[0058] In some example embodiments, the bitmap checker BC may generate a first thread and a second thread to perform a search of the bitmap (e.g., Figure 4 the BM shown).

[0059] The first thread may call the prefetch requester PFR such that, for the bitmap (e.g., Figure 4The user data 132a corresponding to the bit value 1 in the BM shown (which indicates that the user data 132a is dirty data) can be prefetched into the prefetch data area 133c of the buffer memory 133.

[0060] The second thread can call the cache flush requester CFR to search for the latest data of the data corresponding to the bitmap (e.g., Figure 4 the BM shown), regardless of whether the bit value of the bitmap (e.g., Figure 4 the BM shown) is 1 or 0.

[0061] The prefetch requester PFR can prefetch the user data 132a in the non-volatile memory 132 corresponding to the bit value 1 of the bitmap (e.g., Figure 4 the BM shown) (which indicates that the user data is dirty data) into the buffer memory 133.

[0062] The cache flush requester CFR can transmit or send a request command to the CXL switch 120a to request other devices connected to the CXL switch 120a to perform cache flushing of the data corresponding to the bitmap (e.g., Figure 4 the BM shown). Then, in some example embodiments, in response to receiving the request command, the CXL switch 120a can request cache flushing from other devices connected to the CXL switch 120a.

[0063] In some example embodiments, the prefetch operation of the prefetch requester PFR can be performed in a manner overlapping with the operation of transmitting or sending the request command for the cache flush requester CFR. For example, in some example embodiments, when performing the operation of transmitting or sending the request command for the cache flush requester CFR, the prefetch operation of the prefetch requester PFR can be performed as a background operation.

[0064] The ECC engine 131e can detect and correct errors in the data stored in the non-volatile memory 132. For example, the ECC engine 131e can generate parity bits for the user data 132a to be stored in the non-volatile memory 132, and can store the generated parity bits together with the user data 132a in the non-volatile memory 132. For example, when reading the user data 132a from the non-volatile memory 132, the ECC engine 131e can use the user data 132a read from the non-volatile memory 132 and the parity bits read together to detect and correct errors in the user data 132a.

[0065] The buffer memory interface circuit 131f may control the buffer memory 133 such that data is stored in the buffer memory 133 or data is read from the buffer memory 133. In some example embodiments, the buffer memory interface circuit 131f may be implemented to conform to standards such as DDR (Double Data Rate) interface, Low Power Double Data Rate (LPDDR) interface, etc.

[0066] Under the control of the CXL memory controller 131, data may be stored in the buffer memory 133 or the stored data may be output from the buffer memory 133. In addition, in some example embodiments, various information required for the operation of the CXL memory device 130 may be stored in the buffer memory 133.

[0067] In some example embodiments, the buffer memory 133 may be a high-speed memory such as DRAM. However, example embodiments of the inventive concept are not limited thereto.

[0068] Reference Figure 2 and Figure 4 , in some example embodiments, the buffer memory 133 may include a shared memory area 133a that is a host-managed device memory (HDM), and a prefetch data area 133c that stores prefetch data PFD fetched from the non-volatile memory 132, and the prefetch data area 133c is not an HDM (e.g., different from the HDM). For example, the HDM is a memory area existing inside the CXL device 130 and may be accessed by the HA (e.g., host CPU) or other CXL devices (e.g., other CXL memory devices, accelerators, etc.) at its physical address.

[0069] The shared memory area 133a may store therein data collected by the CXL memory controller 131 that monitors its state (e.g., the state of the CXL memory device 130, such as its hardware state and software state). The hardware state may include the remaining capacity, the number of bad blocks, temperature, lifespan, etc., but example embodiments are not limited thereto. The software state may include the busy level, the amount of commands (or requests) received from the host 101, the command pattern (or request pattern) frequently requested from the host 101, and the data pattern requested from the host.

[0070] In some example embodiments, a bitmap BM (which indicates whether the user data 132a stored in the non-volatile memory 132 is dirty data) may be stored in the shared memory area 133a.

[0071] Reference Figure 5 , in some example embodiments, each bit value (0 or 1) of the bitmap BM may indicate whether the user data 132a stored in the non-volatile memory 133 is dirty data.

[0072] In some example embodiments, one bit value of the bitmap BM may be set for each page data of the non-volatile memory 133. For example, in the bitmap BM, one bit value may be set based on 4KB data of the non-volatile memory 133. However, the example embodiments of the inventive concept are not limited thereto. In some example embodiments, in the bitmap BM, one bit value may be set based on 64 bytes or 512 bytes or 2MB data of the non-volatile memory 133.

[0073] Reference Figure 5 , in some example embodiments, the bit value 0 of the bitmap BM may indicate that the corresponding user data UD1, UD3, and UD5 or the data of the corresponding page is not dirty data. The bit value 1 of the bitmap BM may indicate that the corresponding user data UD2 and UD4 or the data of the corresponding page is dirty data.

[0074] The bit values of the bitmap BM may be managed by the dirty tracker DT of the storage controller 131, as previously described according to some example embodiments. However, the example embodiments of the inventive concept are not limited thereto.

[0075] Return reference Figure 1 , in the computing system 100, other devices connected to the CXL storage device 130 via the CXL interface 120 may access the shared memory area 133a of the CXL storage device 130. For example, a CXL storage controller (e.g., Figure 1 the CXL storage controller 141 shown) may access the shared memory area 133a of the CXL storage device 130 via the CXL switch 120a. A host (e.g., Figure 1 the host 101 shown) may access the shared memory 133a of the CXL storage device 130 via the CXL switch 120a.

[0076] Reference Figure 2 , the shared memory area 133a is shown as being set in / within the buffer memory 133. However, the example embodiments of the inventive concept are not limited thereto. For example, in some example embodiments, the shared memory area 133a may be set outside the buffer memory 133 to be accessible by the CXL storage controller 131. In other words, the data stored in the shared memory area 133a may not be the data temporarily stored in the buffer memory 133, but in some example embodiments, may be data non-temporarily (e.g., permanently) stored in the CXL storage device 130 (e.g., in the CXL storage controller 131) for the operation of the CXL storage device 130.

[0077] Data prefetched by a prefetcher PFR of a storage controller 131 from a nonvolatile memory 132 may be stored in a prefetched data area (PFDA) 133c. For example, user data 132a among the user data 132a of the nonvolatile memory 132 corresponding to a bit value 1 of a bitmap BM may be prefetched by the prefetcher PFR from the nonvolatile memory 132 and then may be stored in the prefetched data area 133c.

[0078] According to some example embodiments, a NAND interface circuit 133b may control the nonvolatile memory 132 such that data is stored in the nonvolatile memory 132 or data is read from the nonvolatile memory 132. In some example embodiments, the NAND interface circuit 133b may be implemented to conform to a standard such as a toggle interface or an ONFI (Open NAND Flash Interface Working Group).

[0079] For example, when the nonvolatile memory 132 includes a plurality of NAND flash devices and the NAND interface circuit 133b is implemented based on a toggle interface, the NAND interface circuit 133b may communicate with the plurality of NAND flash devices via a plurality of channels, and the plurality of NAND flash devices may be connected to the plurality of channels via a multi-channel multiplex structure.

[0080] In some example embodiments, the NAND interface circuit 133b may transmit or send a chip enable signal / CE, a command latch enable signal CLE, an address latch enable signal ALE, a read enable signal / RE, and a write enable signal / WE to each of the plurality of NAND flash devices via each of the plurality of channels. Further, in some example embodiments, the NAND interface circuit 133b and each of the plurality of NAND flash devices may exchange a data signal DQ and a data strobe signal DQS with each other via each of the plurality of channels.

[0081] In Figure 2 , the NAND interface circuit 133b is shown as being included in a buffer memory 133. However, example embodiments of the inventive concept are not limited thereto. For example, when the buffer memory 133 is included in the storage controller 131, a NAND interface (I / F) circuit 133b may be disposed inside the storage controller 131 and outside the buffer memory 133.

[0082] Under the control of the CXL memory controller 131, user data 132a can be stored in the non-volatile memory 132 or output from the non-volatile memory 132. In some example embodiments, the non-volatile memory 132 may include a host-managed device memory (HDM) that stores user data 132a. In some example embodiments, the non-volatile memory 132 may include a NAND flash memory. However, the example embodiments of the inventive concept are not limited thereto. In some example embodiments, the non-volatile memory 132 may include non-volatile memories other than NAND flash memories, such as phase change RAM (PRAM), resistive RAM (RRAM), magnetic RAM (MRAM), etc.

[0083] Figure 6 is a schematic diagram showing according to some example embodiments of Figure 2 an example block diagram of the non-volatile memory in.

[0084] Referring to Figure 6 , the non-volatile memory 132 may include a control logic circuit 510, a memory cell array 520, a page buffer unit 550, a voltage generator 530, and a row decoder 540. Although Figure 6 not shown in, in some example embodiments, the non-volatile memory 132 may further include an interface circuit. In addition, in some example embodiments, the non-volatile memory 132 may include column logic, a pre-decoder, a temperature sensor, a command decoder, an address decoder, etc., but the example embodiments are not limited thereto.

[0085] The control logic circuit 510 may control all kinds of operations within the non-volatile memory 132. The control logic circuit 510 may output various control signals in response to commands CMD and / or addresses ADDR from the memory interface circuit 131f. For example, the control logic circuit 510 may output a voltage control signal CTRL_vol, a row address X-ADDR, and a column address Y-ADDR.

[0086] The memory cell array 520 may include a plurality of memory blocks BLK1 to BLKz, where z is a positive integer. Each of the plurality of memory blocks BLK1 to BLKz may include a plurality of memory cells. The memory cell array 520 may be connected to the page buffer unit 550 via bit lines BL, and may be connected to the row decoder 540 via word lines WL, string selection lines SSL, and ground selection lines GSL.

[0087] In some example embodiments, the memory cell array 520 may include a three-dimensional memory cell array. The three-dimensional memory cell array may include a plurality of NAND strings. Each NAND string may include memory cells respectively connected to word lines WL vertically disposed on / located on a substrate. In addition, it will be understood that when an element is referred to as being "on" another element, it may be directly on the other element, or there may also be an intermediate element. In contrast, when an element is referred to as being "directly on" another element, there is no intermediate element. It will also be understood that when an element is referred to as being "on" another element, it may be above or below the other element or adjacent to the other element (e.g., horizontally adjacent). In some example embodiments, the memory cell array 520 may include a two-dimensional memory cell array. The two-dimensional memory cell array may include a plurality of NAND strings arranged in a row direction and a column direction.

[0088] The page buffer unit 550 may include a plurality of page buffers PB1 to PBn, where n is an integer greater than or equal to 3. The plurality of page buffers PB1 to PBn may be respectively connected to the memory cells via a plurality of bit lines BL. The page buffer unit 550 may select at least one bit line from among the bit lines BL in response to a column address Y-ADDR. The page buffer unit 550 may act as and / or be used as a write driver or a read amplifier according to an operation mode. For example, in a programming operation, the page buffer unit 550 may apply a bit line voltage corresponding to the data to be programmed to the selected bit line. In a read operation, the page buffer unit 550 may detect or determine the current or voltage of the selected bit line BL, and thereby detect or determine the data stored in the memory cell based on the detected or determined current or voltage.

[0089] The voltage generator 530 may generate various voltages for performing programming, reading, and erasing operations based on a voltage control signal CTRL_vol. For example, the voltage generator 530 may generate a programming voltage, a read voltage, a programming verification voltage, an erase voltage, etc. as a word line voltage VWL.

[0090] The row decoder 540 may select one of a plurality of word lines WL in response to a row address X-ADDR, and may select one of a plurality of string select lines SSL. For example, in a programming operation, the row decoder 540 may apply a programming voltage and a programming verification voltage to the selected word line, and may apply a read voltage to the selected word line in a read operation.

[0091] Figure 7 is a diagram showing a 3D V-NAND structure that can be applied to a non-volatile memory according to some example embodiments. For example, when the memory module of a storage device is implemented as a 3D V-NAND type flash memory, each of a plurality of memory blocks constituting the memory module may be represented by an equivalent circuit as shown in Figure 7 shown.

[0092] Figure 7 The illustrated memory block BLKi represents a three-dimensional memory block formed in a three-dimensional structure on a substrate. For example, a plurality of memory NAND strings included in the memory block BLKi may extend in a direction perpendicular to the substrate.

[0093] Reference Figure 7 , the memory block BLKi may include a plurality of memory NAND strings NS11 to NS33 disposed between / and located at bit lines BL1, BL2, BL3 and a common source line CSL and connected to the bit lines BL1, BL2, BL3 and the common source line CSL. Each of the plurality of memory NAND strings NS11 to NS33 may include a string select transistor SST, a plurality of memory cells MC1, MC2... MC8, and a ground select transistor GST. In Figure 7 , an example in which each of the plurality of memory NAND strings NS11 to NS33 includes eight memory cells MC1, MC2... MC8 is shown. However, example embodiments of the inventive concept are not necessarily limited thereto.

[0094] The string select transistor SST may be connected to a corresponding one of string select lines SSL1, SSL2, and SSL3. The plurality of memory cells MC1, MC2... MC8 may be respectively connected to corresponding gate lines GTL1, GTL2... GTL8. The gate lines GTL1, GTL2... GTL8 may serve as word lines. Some of the gate lines GTL1, GTL2... GTL8 may serve as dummy word lines. The ground select transistor GST may be connected to a corresponding one of ground select lines GSL1, GSL2, and GSL3. The string select transistor SST may be connected to a corresponding one of the bit lines BL1, BL2, and BL3, and the ground select transistor GST may be connected to the common source line CSL.

[0095] Word lines (e.g., WL1) at the same vertical height may be integrated into one word line. The ground select lines GSL1, GSL2, and GSL3 at the same vertical height may be separated from each other. The string select lines SSL1, SSL2, and SSL3 at the same vertical height may be separated from each other. Figure 7 An example in which the memory block BLK is connected to eight gate lines GTL1, GTL2... GTL8 and three bit lines BL1, BL2, and BL3 is shown. However, example embodiments of the inventive concept are not necessarily limited thereto. Further, it will be understood that an element and / or its property may be described herein as "the same" or "equal" to other elements, and it will also be understood that an element and / or its property may be described herein as "the same", "equal", or "substantially the same" or "substantially equal" to other elements and / or their properties.

[0096] In the following, with reference to Figures 8 to 14 , the operation of a storage system according to some example embodiments will be described.

[0097] Figure 8 is a flowchart showing the operation of a storage system according to some example embodiments. Figures 9 to 14 is a diagram showing the operation as Figure 8 shown. In addition, it will be understood that, unless otherwise explicitly stated, the order of operations or steps is not limited to the order presented in the claims or the drawings. In some example embodiments, the order of operations or steps may be changed, several operations or steps may be combined, a certain operation or step may be split, or a specific operation or step may not be performed.

[0098] With reference to Figure 8 , in operation S100, the host 101 transmits or sends a transfer command CMD to the CXL switch 120a. Then, in operation S105, the CXL switch 120a transmits or sends the transfer command CMD to the CXL storage device 130.

[0099] With reference to Figure 9 , the transfer command transmitted or sent by the host 101 may be a command for instructing to transfer or send dirty data among the user data 132a stored in the non-volatile memory 132 of the CXL storage device 130 to the non-volatile memory 142 of the CXL storage device 140 in a device-to-device transfer / sending manner. In an embodiment, the CXL storage device 140 may further include a buffer memory 143.

[0100] In some example embodiments, in order to change a virtual machine, the host 101 may transmit or send a transfer command to the CXL storage device 130, which command instructs to transfer or transmit dirty data (e.g., updated data). However, the example embodiments of the inventive concept are not limited thereto.

[0101] When transferring data between devices (e.g., CXL storage devices 130 and 140), several factors may need to be considered or are preferably considered.

[0102] First, for example, when transferring dirty data from the non-volatile memory 132 of the CXL storage device 130 to the non-volatile memory 142 of the CXL storage device 140, the transfer is a transfer between non-volatile memories, such that the data transfer or transmission speed may be low or very low.

[0103] In addition, in some example embodiments, since the computing system 100 is or may be operating continuously when issuing and processing transfer commands (e.g., during the period when the computing system 100 issues and processes transfer commands, simultaneously with the computing system 100 issuing and processing transfer commands, etc.), data updates may occur simultaneously, such that the user data 132a stored in the non-volatile memory 132 of the CXL storage device 130 may be data that does not have the latest state.

[0104] Accordingly, in some example embodiments, when transferring data between devices (e.g., CXL storage devices 130 and 140) without considering the above factors, the data transfer or transmission speed may be low, and data inconsistency may occur. Thus, according to some example embodiments, to increase the data transfer or transmission speed, data transfer or transmission between devices may be performed by prefetching data in a buffer memory 133 having a data transfer or transmission speed higher than that of the non-volatile memory 132. In some example embodiments, during the process of transferring, sending, or transmitting data between devices (e.g., CXL storage devices 130 and 140), a cache flush may be requested for the cached data stored in the cache CM2 of other devices (in this example, taking the accelerator 170 (also shown as a cache device in Figure 8 as an example, but the example embodiments of the inventive concept are not limited thereto)) connected to the CXL switch 120a, such that data with the latest state can be transferred, transmitted, or sent between devices.

[0105] Hereinafter, this operation will be described in more detail according to some example embodiments.

[0106] Referring again to Figure 8 , in some example embodiments, after receiving a transfer command (e.g., in response to receiving a transfer command), in operation S120, the CXL storage device 130 may generate a first thread Th1 and a second thread Th2 for data transfer or transmission operations between devices. In some example embodiments, the storage controller 131 of the CXL storage device 130 may perform this operation. In addition, in some example embodiments, the dirty data transfer manager 131d of the storage controller 131 of the CXL storage device 130 may perform this operation. Hereinafter, an example in which the storage controller 131 of the CXL storage device 130 controls the operation of the CXL storage device 130 will be described according to some example embodiments.

[0107] In operation S130, the first thread Th1 performs a prefetch operation. In operation S140, the second thread Th2 performs an operation to request cache flushing. The operation of the second thread Th2 to request cache flushing may include the following operations: In operation S142, the CXL storage device 130 transmits or sends a request command REQ for requesting cache flushing to the CXL switch 120a; In operation S144, the CXL switch 120a transmits or sends a cache flush request command REQ to another device (e.g., the cache device 170) connected to the CXL switch 120a and performing cache operations; In operation S146, the other device (e.g., the cache device 170) performing cache operations transmits or sends the flushed cache data CDATA to the CXL switch 120a; and In operation S148, the CXL switch 120a transmits or sends the flushed cache data CDATA to the CXL storage device 130.

[0108] See Figure 10 , according to some example embodiments, the operations performed by the first thread Th1 and the operations performed by the second thread Th2 may be performed in parallel or substantially in parallel. In some example embodiments, when the second thread Th2 performs the operation of transmitting or sending the request command REQ, the first thread Th1 may perform a prefetch operation as a background operation.

[0109] For example, the first thread Th1 may refer to the bitmap BM1 and perform a prefetch operation PF of user data corresponding to the bit value 1 indicating that the user data is dirty data.

[0110] Refer to Figure 11 , according to some example embodiments, each of the user data UD2 and the user data UD4 among the user data UD1 to UD5 stored in the non-volatile memory 132 corresponds to the bit value 1. Therefore, the first thread Th1 may perform the following operation: Prefetch the user data UD2 and the user data UD4 into the prefetch data area 133c of the buffer memory 133.

[0111] The second thread Th2 may perform an operation CR of transmitting, conveying, or sending a request command REQ for all user data, regardless of the bit value of the bitmap BM1. The reason why the second thread Th2 may transmit or send a request command for all user data regardless of the bit value of the bitmap BM1 may be that even if the bitmap BM1 of the CXL storage device 130 indicates that specific data is not dirty data, the specific data may have been updated due to events that may occur during data transmission or conveyance between devices.

[0112] Thus, according to some example embodiments, in operation S142, the second thread Th2 may transmit, convey, or send a request command to the CXL switch 120a, the request command requesting to flush all user data from the cache, regardless of the bit value of the bitmap BM1, as Figure 12 shown. Then, for example, in operation S144, the CXL switch 120a may transmit, convey, or send a request command to the accelerator 170 connected to the CXL switch 120a, and may perform a cache operation. In operation S146, the accelerator 170 may transmit, convey, or send cache data CDATA to the CXL switch 120a, and then in operation S148, the CXL switch 120a may transmit, convey, or send the cache data CDATA to the CXL storage device 130.

[0113] In some example embodiments, each of the first thread Th1 and the second thread Th2 may perform operations considering the size of the prefetch data area 133c. The first thread Th1 performs a prefetch operation PF on the user data corresponding to the bit value 1 of the bitmap BM1, the bit value 1 indicating that the data is dirty data. The second thread Th2 performs an operation CR of transmitting, conveying, or sending a request command on all user data, regardless of the bit value of the bitmap BM1. Thus, according to some example embodiments, the amount of data prefetched in the prefetch data area 133c where the corresponding cache data CDATA has not been received may gradually increase over time. In this regard, if operations are continued without considering the size of the prefetch data area 133c, an overflow may occur in the prefetch data area 133c. Thus, according to some example embodiments, each of the first thread Th1 and the second thread Th2 may adjust the timing of performing operations considering the size of the prefetch data area 133c.

[0114] Referring to Figure 8 and Figure 14 In some example embodiments, in operation S150, the CXL storage device 130 transmits, conveys, or sends data (DATA) to the CXL switch 120a for data transfer or conveyance between devices. Then, in operation S155, the CXL switch 120a transmits, conveys, or sends the received data to the CXL storage device 140 to complete data conveyance between devices.

[0115] See Figure 13, according to some example embodiments, when transferring, sending, or transmitting data between devices, the second thread Th2 may check the received cache data CDATA for user data corresponding to the bit value 0 of the bitmap BM1. Then, for example, when there is up-to-date data based on the check result, the second thread Th2 may store the up-to-date data in the internal buffer 131c and may transfer, transmit, or send the changed portion thereof to the CXL switch 120a for data transfer between devices. In an embodiment, in response to the refreshed cache data being up-to-date data, the up-to-date data may be sent to another CXL storage device based on the bitmap.

[0116] According to some example embodiments, the second thread Th2 may check the received cache data CDATA for user data corresponding to the bit value 1 of the bitmap BM1. Then, for example, when there is up-to-date data based on the check result, the second thread Th2 may apply the up-to-date data to the data prefetched in the buffer memory 133 and transfer, transmit, or send the application result to the CXL switch 120a. In some example embodiments, when there is no up-to-date data based on the check result, the second thread Th2 may transfer, transmit, or send the data prefetched in the buffer memory 133 to the CXL switch 120a.

[0117] Due to the configuration as described above, the reliability and speed of the storage device according to some example embodiments can be improved when transferring data between devices.

[0118] Figure 15 is an example diagram showing a data center of a computing system applying some example embodiments.

[0119] The above computing system may be included in the data center DECE as an application server and / or a storage server. According to some example embodiments, the storage system as described above may be applied to each of the application server and / or the storage server.

[0120] The data center DECE may collect various data and provide services and may be referred to as a data storage center. For example, the data center DECE may be a system for operating a search engine and a database, or may be a computing system used by a company such as a bank or a government agency, but the example embodiments are not limited thereto and may be a system used by any number and type of companies.

[0121] As Figure 15As shown, in some example embodiments, the data center DECE may include application servers 50_1 to 50_n and storage servers 60_1 to 60_m (each of m and n is an integer greater than 1). The number n of application servers 50_1 to 50_n and the number m of storage servers 60_1 to 60_m may be selected in various ways according to various example embodiments. The number n of application servers 50_1 to 50_n and the number m of storage servers 60_1 to 60_m may be different from each other.

[0122] The application servers 50_1 to 50_n may include at least one of processors 51_1 to 51_n, memories 52_1 to 52_n, switches 53_1 to 53_n, NICs (Network Interface Controllers) 54_1 to 54_n, and storage devices 55_1 to 55_n.

[0123] The processors 51_1 to 51_n may control the overall operations of the application servers 50_1 to 50_n, and may access the memories 52_1 to 52_n and execute instructions and / or data loaded in the memories 52_1 to 52_n. The memories 52_1 to 52_n may include, for example, DDR SDRAM (Double Data Rate Synchronous DRAM), HBM (High Bandwidth Memory), HMC (Hybrid Memory Cube), DIMM (Dual In-line Memory Module), Optane DIMM, or NVMe DIMM (Non-Volatile DIMM), but the example embodiments are not limited thereto.

[0124] In some example embodiments, the number of processors and memories included in the application servers 50_1 to 50_n may be selected in various ways. In some example embodiments, the processors 51_1 to 51_n and the memories 52_1 to 52_n may provide a processor-memory pair. In some example embodiments, the number of processors 51_1 to 51_n and the number of memories 52_1 to 52_n may be different from each other. The processors 51_1 to 51_n may include single-core processors or multi-core processors. In some example embodiments, as shown by the dashed line, the storage devices 55_1 to 55_n in the application servers 50_1 to 50_n may be omitted. The number of storage devices 55_1 to 55_n included in the storage servers 50_1 to 50_n may be selected in various ways according to various example embodiments.

[0125] The processors 51_1 to 51_n, the memories 52_1 to 52_n, the switches 53_1 to 53_n, the NICs 54_1 to NIC 54_n, and / or the storage devices 55_1 to 55_n may communicate with each other via a CXL interface and a CXL switch, as previously described according to some example embodiments.

[0126] The storage servers 60_1 to 60_m may include at least one of processors 61_1 to 61_m, memories 62_1 to 62_m, switches 63_1 to 63_m, NICs 64_1 to NIC 64_n, and storage devices 65_1 to 65_m. The processors 61_1 to 61_m and the memories 62_1 to 62_m may operate similarly to the processors 51_1 to 51_n and the memories 52_1 to 52_n of the application servers 50_1 to 50_n as described above.

[0127] The application servers 50_1 to 50_n and the storage servers 60_1 to 60_m may communicate with each other via the network 70. In some example embodiments, the network 70 may be implemented using Fibre Channel (FC) or Ethernet. FC may be a medium for higher-speed data transmission. An optical switch providing high performance / high availability may be used as FC. Depending on the access scheme of the network 70, the storage servers 60_1 to 60_m may be provided as file storage, block storage, or object storage.

[0128] In some example embodiments, the network 70 may be implemented as a storage-specific network such as a SAN (Storage Area Network). For example, the SAN may be an FC-SAN, which uses an FC network and is implemented according to the FCP FC protocol. In some example embodiments, the SAN may be an IP-SAN, which uses a TCP / IP network and is implemented according to the iSCSI (SCSI over TCP / IP or Internet SCSI) protocol. In some example embodiments, the network 70 may be a general network such as a TCP / IP network. For example, the network 70 may be implemented according to protocols such as FCoE (FC over Ethernet), NAS (Network Attached Storage), and NVMe-oF (NVMe over Fabrics).

[0129] Hereinafter, the application server 50_1 and the storage server 60_1 will be mainly described by way of example. The description of the application server 50_1 may also be applicable to other application servers (e.g., 50_n). The description of the storage server 60_1 may also be applicable to other storage servers (e.g., 60_m).

[0130] When receiving a request to store data from a user or a client, the application server 50_1 may store the data in one of the storage servers 60_1 to 60_m via the network 70. In addition, when receiving a request to read data from a user or a client, the application server 50_1 may obtain the data from one of the storage servers 60_1 to 60_m via the network 79. For example, the application server 50_1 may be implemented as a Web server or a DBMS (Database Management System).

[0131] The application server 50_1 can access the memories 52_n and / or storage devices 55_n included in other application servers 50_n via the network 70, and / or can access the memories 62_1 to 62_m and / or storage devices 65_1 to 65_m respectively included in the storage servers 60_1 to 60_m via the network 70. Thus, the application server 50_1 can perform various operations on the data stored in the application servers 50_1 to 50_n and / or the storage servers 60_1 to 60_m. For example, the application server 50_1 can execute instructions for moving or copying data between the application servers 50_1 to 50_n and / or the storage servers 60_1 to 60_m. At this time, the data can be transmitted from the storage devices 65_1 to 65_m of the storage servers 60_1 to 60_m to the memories 52_1 to 52_n of the application servers 50_1 to 50_n via the memories 62_1 to 62_m of the storage servers 60_1 to 60_m, or the data can be directly transmitted from the storage devices 65_1 to 65_m of the storage servers 60_1 to 60_m to the memories 52_1 to 52_n of the application servers 50_1 to 50_n. In some example embodiments, for security or privacy reasons, the data flowing through the network 70 can be encrypted data.

[0132] In the storage server 60_1, the interface IF can provide a physical connection between the processor 61_1 and the controller CTRL and a physical connection between the NIC 64_1 and the controller CTRL. For example, the interface IF can be implemented in a DAS (Direct Attached Storage) scheme, in which the storage device 65_1 is directly connected to a dedicated cable. In addition, for example, the interface IF can be implemented in various interface schemes, such as ATA (Advanced Technology Attachment), SATA (Serial ATA), e-SATA (External SATA), SCSI (Small Computer System Interface), SAS (Serial Attached SCSI), PCI (Peripheral Component Interconnect), PCIe (PCI Express), NVMe (NVM Express), IEEE 1394, USB (Universal Serial Bus), SD (Secure Digital) card, MMC (Multimedia Card), eMMC (Embedded Multimedia Card), UFS (Universal Flash Storage), eUFS (Embedded Universal Flash Storage), and CF (CompactFlash) card interfaces, etc., but the example embodiments are not limited thereto.

[0133] In the storage server 60_1, the switch 63_1 can selectively connect the processor 61_1 and the storage device 65_1 to each other under the control of the processor 61_1, or can selectively connect the NIC 64_1 and the storage device 65_1 to each other.

[0134] In some example embodiments, NIC 64_1 may include a network interface card, a network adapter, etc. NIC 54_1 may be connected to network 70 via a wired interface, a wireless interface, a Bluetooth interface, an optical interface, etc. NIC 54_1 may include an internal memory, a DSP, a host bus interface, etc., and may be connected to processor 61_1 and / or switch 63_1, etc. via the host bus interface. In some example embodiments, NIC 64_1 may be integrated with at least one of processor 61_1, switch 63_1, and storage device 65_1.

[0135] In application servers 50_1 to 50_n or storage servers 60_1 to 60_m, processors 51_1 to 51_m or 61_1 to 61_n may transmit, convey, or send commands to storage devices 55_1 to 55_n or 65_1 to 65_m or memories 52_1 to 52_n or 62_1 to 62_m to program or read data therefrom. In some example embodiments, the data may be data error-corrected by an ECC (error correction code) engine. The data may be data that has undergone DBI (data bus inversion) or DM (data masking), and may include CRC (cyclic redundancy code) information. For security or privacy, the data may be encrypted data.

[0136] Storage devices 55_1 to 55_n or 65_1 to 65_m may transmit, convey, or send control signals and command / address signals to a non-volatile storage device NVM (e.g., a NAND flash device) in response to read commands received from processors 51_1 to 51_m or 61_1 to 61_n. Thus, in some example embodiments, when reading data from the non-volatile storage device NVM, a read enable signal may be input as a data output control signal for allowing the data to be output to the DQ bus. The read enable signal may be used to generate a data strobe signal. The command and address signals may be latched according to the rising edge or falling edge of the write enable signal.

[0137] The controller CTRL can control all operations of the storage device 65_1. In some example embodiments, the controller CTRL can include SRAM (Static Random Access Memory). The controller CTRL can write data to the non-volatile memory device NVM in response to a write command. Alternatively, in some example embodiments, the controller CTRL can read data from the non-volatile memory device NVM in response to a read command. For example, write commands and / or read commands can be generated based on requests provided from a host (such as processors 61_1 in storage server 60_1, processors 61_m in other storage servers 60_m, or processors 51_1 to 51_n in application servers 50_1 to 50_n). The buffer BUF can temporarily store (buffer) data to be written to the non-volatile memory device NVM or data read from the non-volatile memory device NVM. In some example embodiments, the buffer BUF can include DRAM. Additionally, the buffer BUF can store metadata therein. The metadata can refer to data generated by the controller CTRL to manage user data or data of the non-volatile memory device NVM. The storage device 65 can include a security element (SE) for security or privacy.

[0138] Although some example embodiments of the inventive concept have been described above with reference to the accompanying drawings, the inventive concept is not limited to the described example embodiments and can be implemented in various different forms. Those of ordinary skill in the art to which the inventive concept pertains and the technical field will be able to understand that the inventive concept can be implemented in other specific forms without changing the technical idea or features of the inventive concept. Therefore, it should be understood that the example embodiments described above are illustrative in all aspects and not restrictive.

Claims

1. A storage device, comprising: A first memory; A second memory configured to store a bitmap indicating whether data stored in the first memory is dirty data; And A storage controller configured to control the first memory and the second memory, the storage controller being configured to: Receive a transfer command from a Compute Express Link (CXL) switch to send dirty data stored in the first memory to a first external device connected to the CXL switch; And In response to the transfer command, Prefetch the dirty data stored in the first memory into the second memory based on the bitmap; And Send a request command to the CXL switch to request cache flushing from at least one second external device connected to the CXL switch.

2. The storage device according to claim 1, wherein, The storage controller is configured to perform the prefetch as a background operation while sending the request command.

3. The storage device according to claim 2, wherein, The storage controller is configured to: In response to the transfer command, Generate a first thread and a second thread, The first thread being configured to perform the prefetch based on the bitmap, and The second thread being configured to send the request command regardless of the bitmap.

4. The storage device according to claim 3, wherein, The second memory includes: A host-managed device memory (HDM) area configured to store the bitmap; and A prefetch data area configured to store data prefetched from the first memory.

5. The storage device according to claim 4, wherein, The first thread and the second thread are respectively configured to perform the prefetch and send the request command based on the size of the prefetch data area.

6. The storage device according to claim 2, wherein, The first memory includes non-volatile memory, and the second memory includes volatile memory.

7. The storage device according to claim 6, wherein, The first memory includes NAND flash memory, and the second memory includes dynamic random access memory (DRAM).

8. The storage device according to claim 1, wherein, The storage controller is configured to: Not perform the prefetch when the bitmap indicates that the data stored in the first memory is not dirty data; or Perform the prefetch when the bitmap indicates that the data stored in the first memory is dirty data.

9. The storage device according to claim 8, wherein, Send the request command regardless of the bitmap.

10. The storage device according to claim 1, wherein, The storage controller is configured to: Check the cache data refreshed and received from the at least one second external device in response to the request command; And In response to the refreshed cache data being up-to-date data, send the up-to-date data to the first external device based on the bitmap.

11. The storage device according to claim 10, further comprising a third memory, and the storage controller is configured to: When the bitmap indicates that the data stored in the first memory is not dirty data, Store the up-to-date data in the third memory, and Send the up-to-date data stored in the third memory to the first external device; or When the bitmap indicates that the data stored in the first memory is dirty data, Store the data prefetched in the second memory into the third memory, Update the data stored in the third memory based on the up-to-date data, and Send the up-to-date data stored in the third memory to the first external device.

12. The storage device according to claim 11, wherein, The first memory includes NAND flash memory, the second memory includes dynamic random access memory (DRAM), and the third memory includes static random access memory (SRAM).

13. The storage device according to claim 11, wherein, The third memory is inside the storage controller.

14. The storage device according to claim 1, wherein, The first memory includes a host - managed device memory (HDM) area managed by a first host indicated by the bitmap, The second memory includes: a second HDM area configured to store the bitmap; and a pre - fetched data area configured to store data pre - fetched from the first memory.

15. A method for operating a storage device, the storage device including: a first memory including a host - managed device memory (HDM) area; a second memory, the second memory including: a second HDM area configured to store a bitmap, the bitmap being configured to store bit values corresponding to data stored in the first memory, and a pre - fetched data area configured to store data pre - fetched from the first memory; and a storage controller configured to control the first memory and the second memory, the method including: receiving, by the storage controller, a transfer command from a Compute Express Link (CXL) switch, the transfer command being configured to indicate sending data corresponding to a first bit value of the bitmap among the data stored in the first memory to a first external device connected to the CXL switch; in response to the transfer command, pre - fetching, by the storage controller, data corresponding to the first bit value of the bitmap from the first memory to the second memory; and in response to the transfer command, sending, by the storage controller, a request command to the CXL switch, the request command being configured to request cache flushing of data corresponding to the first bit value and a second bit value of the bitmap from at least one second external device connected to the CXL switch.

16. The method for operating the storage device according to claim 15, wherein, The first bit value is configured to indicate that the data stored in the first memory is dirty data, and the second bit value is configured to indicate that the data stored in the first memory is not dirty data.

17. The method for operating the storage device according to claim 15, further comprising: Generating, by the storage controller, a first thread and a second thread in response to the transfer command, the first thread being configured to: execute the sending of the request command, and the second thread being configured to: perform the pre - fetching as a background operation while the sending of the request command is being executed.

18. The method for operating the storage device according to claim 15, further including: checking, by the storage controller, cache data refreshed and received from the at least one second external device in response to the request command; and in response to the refreshed cache data being up - to - date data, sending, by the storage controller, the up - to - date data to the first external device based on the bitmap.

19. The method for operating the storage device according to claim 18, wherein, The storage device further includes a third memory, and the method for operating the storage device further includes: in response to the bitmap including the first bit value, storing, by the storage controller, the data pre - fetched in the second memory into the third memory, The storage controller updates the data stored in the third memory based on the latest data, and the storage controller sends the latest data stored in the third memory to the first external device; and in response to the bitmap including the second bit value, the storage controller stores the latest data in the third memory, and the storage controller sends the latest data stored in the third memory to the first external device.

20. A storage system, comprising: a host device; a first Compute Express Link (CXL) device; a second CXL device; and a CXL switch configured to connect the host device to the first CXL device and the second CXL device to each other via a CXL interface, wherein the first CXL device includes: a first memory; a second memory configured to store a bitmap indicating whether the data stored in the first memory is dirty data; and a storage controller configured to control the first memory and the second memory, the storage controller being configured to: receive a transfer command from the CXL switch to send data indicated as dirty data by the bitmap among the data stored in the first memory to the second CXL device; and in response to the transfer command, prefetch the data indicated as dirty data stored in the first memory to the second memory based on the bitmap, and send a request command to the CXL switch, the request command requesting a cache flush of data related to the transfer command.

Images