Storage device and method of operating the same

Abstract

A storage device and an operating method are provided. The storage device includes at least one non-volatile memory including a single-level cell region and a multi-level cell region; and a storage controller configured to dynamically resize the single-level cell region through reinforcement learning.

Description

[0001] Cross-reference to related applications

[0002] This patent application claims priority to Korean Patent Application No. 10-2019-0143439, filed on November 11, 2019, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0003] This disclosure relates to a storage device, and more specifically, to a storage device utilizing reinforcement learning.

[0004] Recently, portable electronic devices such as digital cameras, MP3 players, mobile phones, and personal digital assistants (PDAs) have been widely used. Non-volatile memory, such as flash memory, is mainly used in these portable electronic devices due to its low power consumption and high integration.

[0005] Flash memory comprises multiple storage blocks, each of which is divided into single-level cell regions or multi-level cell regions. When flash memory stores write data received from the host, it stores the write data in a single-level cell region or a multi-level cell region storage block according to the characteristics of the write data. Summary of the Invention

[0006] At least one example embodiment of this disclosure provides a storage device with improved performance.

[0007] At least one example embodiment of the present invention provides a storage device that can flexibly and dynamically adjust the size of a single-level cell region according to the environment of the storage system.

[0008] According to an example embodiment of this disclosure, a storage device includes: at least one non-volatile memory, the at least one non-volatile memory including a single-level cell region and a multi-level cell region; and a storage controller configured to dynamically resize the single-level cell region through reinforcement learning.

[0009] According to an example embodiment of this disclosure, an operation method for a storage device is provided. The method includes: dividing a non-volatile memory into single-level cell regions and multi-level cell regions based on a threshold segment count value; performing reinforcement learning on environmental information to determine a reward direction when the total amount of write data from a host for writing to the storage device reaches the threshold segment count value; and dynamically adjusting the threshold segment count value according to the reward direction.

[0010] According to an example embodiment of this disclosure, a storage device includes: a non-volatile memory including a single-level cell region and a multi-level cell region; and a storage controller including a processing unit configured to perform reinforcement learning on environmental information of the storage device to determine an optimal ratio of the single-level cell region to the multi-level cell region, and the storage controller dynamically adjusting the size of the single-level cell region according to the determined optimal ratio. Attached Figure Description

[0011] This disclosure will become more apparent from the detailed description of exemplary embodiments thereof with reference to the accompanying drawings, in which:

[0012] Figure 1 This is a block diagram schematically illustrating an example embodiment of a storage system based on the concept of the present invention.

[0013] Figure 2 This illustrates an example embodiment of the concept according to the present invention. Figure 1 Block diagram of the storage controller.

[0014] Figure 3 This illustrates an example embodiment of the concept according to the present invention. Figure 1 A block diagram of non-volatile memory.

[0015] Figure 4 This is a diagram illustrating the configuration of a non-volatile memory according to an exemplary embodiment of the present invention.

[0016] Figure 5 and Figure 6 A flowchart illustrating an explanatory method for operating a storage device according to an exemplary embodiment of the present invention is shown.

[0017] Figure 7 This is a diagram illustrating the cycle of reinforcement learning according to an exemplary embodiment of the present invention.

[0018] Figure 8 This is a diagram illustrating environmental information for reinforcement learning according to an exemplary embodiment of the present invention.

[0019] Figure 9 This is a diagram illustrating an example embodiment of the concept according to the present invention.

[0020] Figure 10 This is a block diagram illustrating a storage controller according to an exemplary embodiment of the present invention.

[0021] Figure 11 This is a block diagram illustrating a storage controller according to an exemplary embodiment of the present invention.

[0022] Figure 12This is a block diagram illustrating an example embodiment of a data storage device according to a concept of the present invention.

[0023] Figure 13 This is a diagram illustrating an example of a storage device being applied to a mobile system according to an example embodiment.

[0024] Figure 14 This is a block diagram illustrating an example embodiment of a universal flash storage (UFS) system based on the present invention.

[0025] Figure 15 This is a block diagram illustrating an example embodiment of a solid-state storage device according to a concept of the present invention. Detailed Implementation

[0026] In the following description, exemplary embodiments of the inventive concept will be described with reference to the accompanying drawings.

[0027] Figure 1 This is a block diagram schematically illustrating an example embodiment of a storage system based on the concept of the present invention. Figure 2 This illustrates an example embodiment of the concept according to the present invention. Figure 1 Block diagram of the storage controller. Figure 3 This illustrates an example embodiment of the concept according to the present invention. Figure 1 A block diagram of non-volatile memory.

[0028] refer to Figure 1 The storage system includes a host 1 (e.g., a host device) and a storage device 1000. The storage device 1000 includes a storage controller 100 (e.g., a memory controller or control circuit) and a non-volatile memory (NVM) 200 (e.g., a flash memory).

[0029] When a write request occurs, host 1 sends the write data and logical address to storage device 1000. According to some embodiments, host 1 may include personal / portable computers, portable electronic devices such as PDAs, personal music players (PMPs) and smartphones, televisions such as high-definition televisions (HDTVs), etc.

[0030] According to some embodiments, the storage device 1000 can be implemented as internal memory embedded in an electronic device, such as an embedded universal flash memory (UFS) storage device, an embedded multimedia card (eMMC), or a solid-state drive (SSD). In some embodiments, the storage device 1000 can be implemented as external memory removable from an electronic device, such as a UFS memory card, a compact flash (CF) card, a secure digital (SD) card, a micro SD card, a mini SD card, a high-speed digital (xD) card, or a memory stick.

[0031] Storage controller 100 connects host 1 to non-volatile memory 200. According to an example embodiment, storage controller 100, in response to a write command received from host 1, programs (e.g., writes) write data provided by host 1 to non-volatile memory 200. Furthermore, storage controller 100 can control read operations on non-volatile memory 200 in response to read commands received from host 1. Additionally, storage controller 100 can perform one or more background operations for non-volatile memory 200, such as wear leveling, garbage collection, and bad block management. For example, wear leveling can ensure priority use of less frequently used storage blocks. For example, garbage collection can move data from valid pages of several storage blocks to a single storage block, and then later perform erase operations on several storage blocks to free up space. For example, bad block management can track bad blocks so that data can be written to or moved from bad blocks to other blocks.

[0032] like Figure 2 As shown, an exemplary embodiment of the storage controller 100 according to the present invention includes a host interface 110 (e.g., interface circuitry), a processing unit 120 (e.g., a processor such as a central processing unit), a memory 130, a register 140, programmable logic 150 (e.g., a field-programmable gate array), and a non-volatile memory interface 160 (e.g., interface circuitry).

[0033] Components in the storage controller 100 are interconnected via a data bus 101. The data bus 101 may include multiple channels. In an example embodiment, the multiple channels may indicate communication paths driven independently of each other, and the multiple channels may communicate based on the same communication method and the devices connected to them.

[0034] Host interface 110 is connected to host 1. In an example embodiment, host interface 110 may be based on at least one of a variety of interfaces, such as Double Data Rate (DDR) interface, Low Power DDR (LPDDR) interface, Universal Serial Bus (USB) interface, Multimedia Card (MMC) interface, Peripheral Component Interconnect (PCI) interface, PCI-express (PCI-E) interface, Advanced Technology Attachment (ATA) interface, Serial ATA (SATA) interface, Parallel ATA (PATA) interface, Small Computer System Interface (SCSI), Enhanced Small Disk Interface (ESDI), Integrated Drive Electronic Device (IDE) interface, Mobile Industrial Processor Interface (MIPI), Non-Volatile Memory Express (NVM-e) interface, and Universal Flash Storage (UFS) interface.

[0035] The processing unit 120 can control the operation of each component in the storage controller 100 and perform arithmetic operations on write commands, read commands, delete commands or other commands received from the host 1 in relation to the operation of the storage device 1000.

[0036] According to an exemplary embodiment of the present invention, processing unit 120 performs learning operations to learn control operations of non-volatile memory 200. In the exemplary embodiment, processing unit 120 performs reinforcement learning operations related to the migration between single-level cell regions and multi-level cell regions.

[0037] The memory 130 may store data necessary for the operation of the memory controller 100. According to some embodiments, the memory 130 may include a cache, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable programmable read-only memory (EPEPROM), a phase-change RAM (PRAM), a flash memory, static RAM (SRAM), or dynamic RAM (DRAM).

[0038] According to some embodiments, memory 130 can store various information related to reinforcement learning, including environmental information or threshold segment count values. In an exemplary embodiment of the invention, memory 130 is a non-volatile memory for storing information such as queue lists.

[0039] Register 140 may be an operation memory used to store write data received from host 1, read data received from non-volatile memory 200, and operation results generated during control operations in processing unit 120. Register 140 may also be referred to as a buffer memory.

[0040] Programmable logic 150 can perform some of the operations performed by processing unit 120. For example, processing unit 120 can offload some of the operations it normally performs to programmable logic 150. For example, processing unit 120 can offload a portion of the operations it normally performs to programmable logic 150. According to an example embodiment, programmable logic 150 is a programmable logic device (PLD) for designing digital circuits that perform specific operations. For example, the PLD may include multiple programmable gate arrays.

[0041] The non-volatile memory interface 160, also referred to as a non-volatile memory controller, accesses the non-volatile memory 200 to control the operation of each of the plurality of non-volatile memories. In an example embodiment, the non-volatile memory interface 160 is connected to the non-volatile memory 200 via at least one channel to write, read, or delete data.

[0042] The non-volatile memory 200 is provided as a storage medium for the storage device 1000. For example, the non-volatile memory 200 can be configured as a NAND flash memory with a large storage capacity.

[0043] In this configuration, the non-volatile memory 200 may include memory regions managed in different ways. Each memory region consists of multiple memory blocks. The non-volatile memory 200 may be configured as NOR flash memory or next-generation non-volatile memory, such as phase-change random access memory (PRAM), magnetic random access memory (MRAM), resistive random access memory (ReRAM), ferroelectric random access memory (FRAM), etc. Alternatively, the non-volatile memory 200 may be replaced by a memory that is volatile memory but is configured entirely as dynamic random access memory (DRAM) or static random access memory (SRAM). Alternatively, the non-volatile memory 200 may be replaced by a memory that includes a non-volatile portion (e.g., flash memory) and a volatile portion (e.g., DRAM).

[0044] like Figure 3 As shown, a non-volatile memory 200 according to an exemplary embodiment of the present invention includes a cell array 201 (e.g., a memory cell array), a row decoder 202 (e.g., a decoder circuit), a page buffer 203, and control logic 204 (e.g., a controller or control circuit).

[0045] Cell array 201 may include multiple memory blocks. For simplicity, Figure 3 A cell array 201 comprising a memory block is shown. Each memory block may consist of multiple pages. Each page may consist of multiple memory cells. In the non-volatile memory 200, erase operations can be performed based on memory blocks, and write or read operations can be performed based on pages.

[0046] Cell array 201 may include multiple memory cells. In an example embodiment, the memory cells have a cell string structure. A cell string includes a string select transistor SST connected to a string select line SSL, multiple memory cells connected to multiple word lines WL0 to WLn-1, and a ground select transistor GST connected to a ground select line GSL. The string select transistor SST is connected to a bit line BL (e.g., one of bit lines BL0, BL1, BL2, ..., BLm-1), and the ground select transistor GST is connected to a common source line CSL.

[0047] The cell array 201 may include a substrate and a plurality of cell strings vertically formed on the substrate. Each of the plurality of cell strings may include a plurality of cell transistors stacked in a direction perpendicular to the substrate. In other words, the cell array 201 may be formed as a three-dimensional structure (or a vertical structure).

[0048] The line decoder 202 is connected to the cell array 201 via select lines SSL and GSL or word lines WL0 to WLn-1. During a programming (e.g., write) or read operation, the line decoder 202 receives an address and selects a word line (e.g., WL1). Furthermore, the line decoder 202 delivers the voltage required for the programming or read operation to the selected or unselected word line.

[0049] Page buffer 203 operates as a write driver or a sense amplifier. Page buffer 203 can temporarily store data to be programmed into or read from a selected memory cell. Page buffer 203 is connected to cell array 201 via bit lines BL0 to BLm-1. In programming operations, page buffer 203 receives data and delivers it to the memory cell of the selected page. In read operations, page buffer 203 reads data from the memory cell of the selected page and outputs the data externally.

[0050] Control logic 204 can control operations such as programming, reading, and erasing of non-volatile memory 200. For example, during a programming operation, control logic 204 can control line decoder 202 to provide programming voltage to the selected word line. Control logic 204 can control page buffer 203 to provide programming data to the selected page.

[0051] Figure 4 This is a diagram illustrating the configuration of a non-volatile memory according to an exemplary embodiment of the present invention.

[0052] In an example embodiment, the non-volatile memory 200 is divided into a first memory region and a second memory region based on a threshold Th. Each memory region may include multiple memory blocks. The first and second memory regions can be managed in different ways. According to the example embodiment, the first memory region is managed in a single-level cell (SLC) mode, and the second memory region is managed in a multi-level cell (MLC) mode. In the example embodiment, some write data is programmed in the first memory region (i.e., the single-level cell region), while the remaining write data, excluding metadata, is programmed in the second memory region (i.e., the multi-level cell region). For example, metadata may be stored in the first memory region. For example, metadata may include mapping information that maps logical addresses from host 1 to physical addresses of the non-volatile memory 200. For example, metadata may include information for wear leveling, such as program / erase counts.

[0053] Programming and reading operations are performed faster in a Single-Level Cell (SLC) area than in a Multi-Level Cell (MLC) area. In an example embodiment, data that changes frequently according to the mode of host 1 is stored in the SLC area, while data that changes infrequently is stored in the MLC area. In an example embodiment, data that changes at a certain rate or higher is stored in the SLC area, while data that changes at a rate lower than the certain rate is stored in the MLC area.

[0054] The storage controller 100 uses a single-level cell region to compensate for the slow performance of multi-level cell regions. Depending on the application approach, the size of the single-level cell region can be adjusted statically or dynamically.

[0055] For example, if the size of a single-level cell region If the size is large, the size of the multilevel cell region (100-Th) decreases by the amount, thereby reducing the space utilization of the non-volatile memory and reducing the overall data storage capacity. In this case, the space utilization can be the ratio of the used area (B) of the multilevel cell region to the entire region (100-Th).

[0056] On the other hand, if the size of the single-level unit region If the size is too small, in order to ensure available space (Th-A) in the single-level cell region, frequent migrations of some data (A) stored in the single-level cell region to multi-level cell regions may occur. Therefore, the performance of the storage device may degrade. Thus, the size of the single-level cell region, i.e., the threshold Th, should be appropriately chosen.

[0057] At least one example embodiment of this disclosure provides a storage device that improves the performance of the storage device by flexibly and dynamically adjusting the size of a single-level cell region according to the environment of the storage system.

[0058] Example embodiments of the present invention Figure 2 The processing unit 120 determines a threshold Th based on a segment count, which is the boundary between the SLC region and the MLC region. That is, the size of the SLC region is determined by the threshold segment count value, and the size of the MLC region is determined as the remaining region (100-Th(%)) obtained by subtracting the SLC region from the entire region (100%) of the non-volatile memory 200. For example, if the non-volatile memory 200 is capable of storing megabytes of data, and the size of the SLC region is resized to store one-tenth of a megabyte based on the threshold segment count value, then the size of the MLC region will be resized to store nine-tenths of a megabyte.

[0059] The processing unit 120 can dynamically change the size of the SLC region (i.e., the threshold segment count value) by performing reinforcement learning operations based on the environmental information of the host 1 and / or storage device 1000.

[0060] Reinforcement learning can be described in terms of agent, action, environment, state, and reward. A reinforcement learning operation is a learning method in which an agent, defined in a specific environment, recognizes the current state and selects an action or a sequence of actions that maximizes the reward among available actions.

[0061] During reinforcement learning operations according to the example embodiment, processing unit 120 (i.e., the agent) performs an action on a threshold segment count value determined in the current period, collects changes in environmental information after the action, and checks the state of storage device 1000 based on the collected information to determine a reward. Processing unit 120 can readjust the threshold segment count value based on the checked state and the determined reward.

[0062] According to an example embodiment, the action is an operation of increasing or decreasing the size of a single-level cell region or a threshold segment count value. For example, increasing the threshold segment count value by a certain amount can result in a bonus such as a 10% increase in the access speed of a memory device. For example, decreasing the threshold segment count value by a certain amount can result in a bonus such as a 30% increase in the storage capacity of a memory device.

[0063] In the example embodiment, the environmental information includes information about host 1, information about non-volatile memory 200, and environmental information about previous periods.

[0064] Storage controller 100 may include a queue list. (See reference...) Figure 9 Provide a detailed description of the status and the list of teams.

[0065] According to an example embodiment, the queue list is stored in memory 130. Alternatively, although not shown, according to some embodiments, the queue list may be stored in external memory connected to processing unit 120. Alternatively, according to some embodiments, the queue list may be stored in non-volatile memory 200.

[0066] In an example embodiment, processing unit 120 determines a reward. The reward can be determined by considering migration costs and the current state of the non-volatile memory. As a reward function, if the migration cost in the current cycle is greater than the average migration cost, the change in the effective / invalid page count ratio of the SLC region between the current cycle and previous cycles can be taken into account to determine whether the reward is calculated positively or negatively. For example, migrating (or moving) a large amount of data from an SLC region to an MLC region due to making the SLC region much smaller may result in a large migration cost, while migrating a small amount of data from an SLC region to an MLC region due to making the SLC region slightly smaller may result in a smaller migration cost.

[0067] If the effective / ineffective page count ratio of the current period is less than that of the previous period, processing unit 120 can calculate a negative reward for the action in the current period. Otherwise, if the effective / ineffective page count ratio of the current period is greater than that of the previous period, processing unit 120 can calculate a positive reward for the action in the current period. Therefore, by considering not only migration costs but also the effective / ineffective page count ratio, the size of the single-level unit area can be prevented from increasing unnecessarily.

[0068] Figure 5 and Figure 6 A flowchart illustrating an explanatory method for operating a storage device according to an exemplary embodiment of the present invention is shown. Figure 7 This is a diagram explaining the cycle of reinforcement learning according to an example embodiment. Figure 8 This is a diagram explaining the environment information for reinforcement learning according to some example embodiments. Figure 9 This is a diagram illustrating the queue list according to some example embodiments.

[0069] refer to Figure 5 When the storage device receives write data from the host (step S10), the storage device checks the characteristics (e.g., features) of the data (step S20).

[0070] If the data is to be written to a single-level cell area, check whether the data can be written to the single-level cell area (step S30). If the data can be written to the single-level cell area (i.e., if the data to be written is less than the empty space of the single-level cell area), the storage device will program (or write) the data to the single-level horizontal cell area (step S40).

[0071] If the data is to be written to a multi-level cell area, the storage device verifies whether the data can be written to the multi-level cell area (step S30) and programs (or writes) the data to the multi-level cell area (step S40).

[0072] However, if the total amount of write data received from the host is greater than the size of the single-level cell region, that is, if there is not enough free space to programmatically write data in the single-level cell region, the boundary between the single-level cell region and the multi-level cell region is adjusted by a threshold predicted by reinforcement learning (step S100).

[0073] A storage device (e.g., an agent) performs an action (step S120) on a non-volatile memory (S110, agent), the non-volatile memory being divided according to a threshold (i.e., a threshold segment count value) set in the current cycle.

[0074] The action can be an operation that increases or decreases the size of a single-level cell region. According to an example embodiment, the action can be an operation that increases or decreases the threshold segment count value.

[0075] The storage device collects environmental information (e.g., state data) about the actions already performed on it (step S130). Environmental information can be collected within a single unit cycle. In other words, reinforcement learning is performed in each unit cycle.

[0076] like Figure 7 As shown, the unit period is determined based on the write count value of data, not time. More specifically, reinforcement learning is performed whenever the total amount of write data received from host 1 is greater than or equal to the size of the single-level cell region, that is, whenever it is no longer possible to program write data into the single-level cell region. Therefore, segments 0-c1 and segments c1-c2 may be different in the time domain, but the same in the count value domain. Here, in this specification, the segment from the point in time when the entire space of the single-level cell region is free (i.e., the free space in the SLC is 100%) to the point in time when the total amount of write data received from host 1 becomes equal to the size of the single-level cell region (i.e., the free space in the SLC is 0%) is called a unit period or epoch. For example, if the free space of the SLC is 100% at time 0, and the free space is 0% after 1 second from time 0, then the epoch is 1 second.

[0077] The environmental information includes environmental information about the host, environmental information about the non-volatile memory, and environmental information about previous periods. In an example embodiment, the environmental information includes environmental information about the host and environmental information about previous periods (i.e., excluding environmental information about the non-volatile memory). The environmental information about the host may be an index of patterns used to identify the host (e.g., an average write segment count value).

[0078] Environmental information about the non-volatile memory 200 may include, for example, space utilization information, valid page count information, invalid page count information, and valid / invalid page count ratio information.

[0079] Space utilization information includes details about the size of free space and used space (space where data has been written) in the current cycle, the space ratio, etc. Although, in Figure 8 In the illustrated embodiment, the space is divided into four segments (each 25%), but according to some embodiments, it can be divided into fewer or more segments. Since write operations of the storage controller 100 are performed based on physical pages, information regarding the effective page count, invalid page count, or effective / invalid page count ratio of the non-volatile memory 200 can also be used as environmental information. Figure 8 In the illustrated embodiment, the valid / invalid page count ratio information is divided into four segments; however, according to some embodiments, it can be divided into fewer or more segments. Furthermore, although the average write segment count is... Figure 8 The illustrated embodiment is divided into six segments, but according to some embodiments, it can also be divided into fewer or more segments. When comparing a previous cycle with a cycle preceding the previous cycle, the previous actions used as historical information for the previous cycle can be divided into three segments: a segment where the state is maintained (maintain); a segment where the size of the single-level cell region has decreased (-SLC block); and a segment where the size of the single-level cell region has increased (+SLC block).

[0080] Information about the environment in previous cycles can include historical information about reinforcement learning in previous cycles, that is, information about the agent, actions, environment, state and rewards on the storage device in previous cycles.

[0081] The storage device checks the status in the queue based on environmental information (step S140). For example... Figure 9 As shown, the queue list can include multiple states defined in reinforcement learning.

[0082] The number of states stored in the queue can be determined based on the factors collected as environmental information and the number of states defined in each factor. As a more concrete example, in Figure 8 In this section, we will describe a scenario where the storage device has four factors as environmental information. Here, we will assume that the storage device has collected previous actions used as historical information for previous cycles, the space utilization of the single-level cell region in the current cycle, the ratio of valid to invalid page counts of the single-level cell region, and the host's average write segment count.

[0083] Storage devices can distinguish multiple states for each factor. (Reference / Based on...) Figure 8 Implementation examples Figure 9 The queue list stores 288 states (4 × 4 × 6 × 3). Specifically, in the example shown, the space utilization of a single-level cell region is divided into 4 states, the ratio of valid to invalid page counts of a single-level cell region is divided into 4 states, the host's average write segment count is divided into 6 states, and previous actions used as historical information for previous cycles are divided into 3 states. Therefore, the total number of states to which the storage device can belong is 288.

[0084] A queue list (e.g., a Q-table) can store the direction of the reward corresponding to each state. The reward function can be determined by a Q-learning equation. The Q-learning equation is one of the reinforcement learning techniques that allows learning without a model. In an example embodiment, the reward function is a function that predicts the expected value of the utility used to perform a given action in a given state. In the Q-learning equation, the resulting value is determined based on the effective / ineffective page count ratio and the data migration cost between single-level and multi-level cell regions, and thus the reward direction can be determined.

[0085] exist Figure 9 In the example shown, if the equation yields a value of 1.04, the action taken by the agent in the current cycle is appropriate, and reinforcement learning is performed by maintaining the current state. If the equation yields a value of -0.32, it is determined that the action taken by the agent in the current cycle has received a negative reward. Therefore, the size of the single-level unit region is reduced, and an action is performed in the direction of reducing the threshold segment count in the next cycle.

[0086] In the example embodiment, Figure 9 Each box in the Q-table is an entry. Therefore, the Q-table can include multiple entries. Each entry can indicate a different one of multiple environmental states. For example, the first state could be a space utilization rate between 0% and 25%, a valid / invalid page count ratio between 0% and 5%, an average write segment count between 16KB and 32KB, and the SLC region size was previously maintained; and the second state could be a space utilization rate between 25% and 50%, a valid / invalid page count ratio between 0% and 5%, an average write segment count between 32KB and 64KB, and the SLC region size was previously increased. In an example embodiment, the storage controller 100 performs reinforcement learning that uses environmental information to determine a selected state from multiple states, calculates the current reward from the selected state and the environmental information, determines a reward direction based on the current reward and the reward history of the entry indicating the selected state, and adjusts the threshold segment count value according to the calculated reward direction.

[0087] According to the example embodiment, when the migration cost of the previous period is greater than or equal to the average migration cost, and the effective / ineffective page count ratio in the current period of a single-level cell region is less than the effective / ineffective page count ratio in its previous period, the reward is determined as a negative reward for adjusting the size of the single-level cell region. That is, the weight of the reward direction is reduced. On the other hand, when the migration cost of the previous period is greater than or equal to the average migration cost, and the effective / ineffective page count ratio in the current period of a single-level cell region is greater than the effective / ineffective page count ratio in its previous period, the reward is determined as a positive reward for increasing the weight of the reward direction.

[0088] Furthermore, according to the example embodiment, when the migration cost of the previous cycle is less than the average migration cost, the reward is determined as a negative reward for readjusting the size of the single-level cell region.

[0089] In addition, when the migration cost in the previous period is less than the average migration cost, and the effective / ineffective page count ratio in the current period of the multi-level cell region is less than the effective / ineffective page count ratio in its previous period, a negative reward is added for resizing the single-level cell region.

[0090] Figure 10 This is a block diagram illustrating a storage controller 2000 according to an exemplary embodiment of the concept of the present invention. For simplicity of description, the main description will be related to... Figure 2 The differences. The storage controller 2000 can be used to implement... Figure 1 Storage controller 100.

[0091] According to the example embodiment, with Figure 2 Unlike other systems, reinforcement learning for the storage device is performed in a separate learning processing unit 170. While the processing unit 120 performs general control operations of the storage controller 100, the learning processing unit 170 performs reinforcement learning on the size of a single-level cell region based on environmental information of the storage device.

[0092] Whenever a single-level cell region is fully used by the host's write data, the learning processing unit 170 can execute a reference. Figure 6 The reinforcement learning described above. For example, the learning processing unit 170 can perform reinforcement learning whenever it is predicted that the written data will completely fill the single-level cell region.

[0093] In an example embodiment, the processing unit 170 is configured to perform reinforcement learning on environmental information to determine the optimal ratio of single-level cell regions to multi-level regions, and the storage controller 100 dynamically adjusts the size of the single-level cell regions according to the determined optimal ratio.

[0094] According to some embodiments, the queue list may be stored in memory 130 or non-volatile memory 200.

[0095] Figure 11 This is a block diagram illustrating a storage controller 2000 according to an exemplary embodiment of the concept of the present invention. For simplicity of description, the main description will be related to... Figure 10 The differences.

[0096] Figure 11 The storage controller 2000 includes Figure 10 The separate learning processing unit 170 is shown. However, compared with... Figure 10 Unlike other memory controllers, the storage controller 2000 also includes a separate external memory 175 for storing information required for reinforcement learning.

[0097] For example, external memory 175 can store information related to team lists and rewards.

[0098] Figure 12 This is a block diagram illustrating an example embodiment of a data storage device according to a concept of the present invention.

[0099] refer to Figure 12 According to an example embodiment of the present disclosure, a data storage device 3000 includes a flash memory chip 3100 and a flash memory controller 3200. The flash memory controller 3200 can control the flash memory chip 3100 based on control signals received from outside the data storage device 3000.

[0100] Configuration of Flash Memory Chip 3100 and Figure 1 , Figure 3 , Figure 4 and Figure 5 The non-volatile memory device 200 is substantially the same as that of the other two and consists of multiple chips. Each flash memory chip 3100 in this embodiment can be formed from any of the following: a multi-layer stacked flash memory structure, a flash memory structure without source-drain, a pin-type flash memory structure, and a three-dimensional flash memory structure.

[0101] The data storage device 3000 according to some embodiments can be configured as a memory card device, a solid-state drive (SSD) device, a multimedia card device, an SD device, a memory stick device, a hard disk drive device, a hybrid drive device, or a universal serial bus flash memory device. For example, the data storage device 3000 of this embodiment can be configured as a memory card that meets industry standards for use in user devices such as digital cameras, personal computers, etc.

[0102] The flash memory controller 3200 can program write data into the flash memory chip 3100. In this case, the flash memory controller 3200 can determine whether the write data is stored in a single-level cell region or a multi-level cell region of the flash memory chip 3100. The flash memory controller 3200 can perform the same functions as the storage controller 100. For example, the flash memory controller 3200 can determine the ratio between the single-level cell region and the multi-level cell region of the flash memory chip 3100 based on the prediction results of the reinforcement learning described above. According to an example embodiment, the flash memory controller 3200 includes a learning processing unit 3205 (e.g., a processor). The learning processing unit 3205 may be... Figure 10 or Figure 11 The learning processing unit.

[0103] Therefore, the data storage device 3000 according to some embodiments can program written data into a single-level cell region or a multi-level cell region based on the prediction results of reinforcement learning, while minimizing the migration cost between the single-level cell region and the multi-level cell region.

[0104] Figure 13 This is a diagram illustrating an example of a storage device being applied to a mobile system according to some embodiments.

[0105] refer to Figure 13 The mobile system 4000 includes a processor 4200, main memory 4300 (e.g., RAM), user interface 4400, modem 4500 such as a baseband chipset, and storage device 4100.

[0106] Processor 4200 can perform various computing functions, such as executing specific software for performing certain calculations or tasks. For example, processor 4200 can be a microprocessor or a central processing unit (CPU). Processor 4200 can be connected to main memory 4300 via bus 4600, such as an address bus, control bus, and / or data bus. For example, main memory 4300 can be implemented as DRAM, mobile DRAM, SRAM, PRAM, FRAM, RRAM, and / or MRAM. Furthermore, processor 4200 can be connected to an expansion bus, such as a peripheral component interconnect (PCI) bus. Therefore, processor 4200 can control user interface 4400, which includes one or more input devices, such as a keyboard or mouse, and one or more output devices, such as a printer or display device. Modem 4500 can wirelessly send / receive data to / from external devices. In non-volatile memory 4120, data processed by processor 4200, data received by modem 4500, etc., can be stored via memory controller 4110. Although not shown, mobile system 4000 may also include a power supply for supplying operating voltage. In addition, according to embodiments, the mobile system 4000 may also include an application chipset, a camera image processor (CI P), etc.

[0107] The memory controller 4110 can perform access operations on the non-volatile memory 4120, namely, data read operations, data write operations and / or data transfer operations.

[0108] The memory controller 4110 may include a reference Figures 1 to 11 The memory controller 100 is described. According to some embodiments, the memory controller 4110 can perform operations related to... Figure 2 The storage controller 100 can perform reinforcement learning-related operations in the learning processing unit 120. According to some embodiments, the storage controller 100 can perform reinforcement learning-related operations in the learning processing unit 4115, which is connected to... Figure 10 or Figure 11 The learning processing unit 170 is the same.

[0109] By enhancing learning, the size of a single-level cell region is dynamically adjusted based on environmental information of the mobile system 4000, thereby dynamically adapting to the current state of the mobile system 4000. Environmental information includes, for example, pattern information of read, write, or delete commands of the mobile system 4000, as well as internal state information or external information of the storage device 4100.

[0110] In some embodiments, the non-volatile memory 4120 may include a single-level cell region and a multi-level cell region 4125.

[0111] Storage device 4100 can be implemented in various types of packages. For example, storage device 4100 can be mounted using packages such as: Package-on-Package (PoP), Ball Grid Array (BGA), Chip-Level Package (CSP), Plastic Leaded Chip Carrier (PLCC), Plastic Dual In-line Package (PDI P), Die-on-Wafer Package (DI WP), Die-on-Wafer Form-of-Factory (DI WF), Chip-on-Board (COB), Ceramic Dual In-line Package (CERDI P), Plastic Metric Square Flat Package (MQFP), Thin Quad Flat Package (TQFP), Small Outline Package (SOP), Shrink Small Outline Package (SSOP), Thin Small Outline Package (TSOP), Thin Quad Flat Package (TQFP), System-in-Package (SIP), Multi-Chip Package (MCP), Wafer-Level Assembly Package (WFP), and Wafer-Level Processing Stack Package (WSP).

[0112] Figure 14 This is a block diagram illustrating an example embodiment of a UFS system based on the concept of the present invention.

[0113] refer to Figure 14 The UFS (Universal Flash Storage) system 5000 includes a UFS host 5100 and a UFS device 5200. The UFS host 5100 and the UFS device 5200 can be connected to each other via a UFS interface 5300. The UFS system 5000 is based on flash memory 5206 as a non-volatile storage device. The UFS system 5000 can be used in mobile devices such as smartphones.

[0114] The UFS host 5100 includes an application 5102, a device driver 5104, a host controller 5106, and a host interface 5108.

[0115] Application 5102 includes various applications running on the UFS host 5100. Device driver 5104 is used to drive peripheral devices connected to the UFS host 5100 and can also drive UFS device 5200. Application 5102 and device driver 5104 can be implemented through software, firmware, etc.

[0116] Host controller 5106 can generate protocols or commands to be provided to UFS device 5200 in response to requests from application 5102 and device driver 5104. The generated commands can be provided to UFS device 5200 via host interface 5108. When host controller 5106 receives a write request from device driver 5104, it provides write commands and data to UFS device 5200 via host interface 5108. When receiving a read request from device driver 5104, host controller 5106 provides read commands to UFS device 5200 via host interface 5108 and receives data from UFS device 5200.

[0117] The UFS interface 5300 uses the Serial Advanced Technology Attachment (SATA) interface. The SATA interface is broadly divided into the physical layer, link layer, and transport layer according to its function.

[0118] The host interface 5108 of the UFS host 5100 includes a transmitter and a receiver, while the device interface 5201 of the UFS device 5200 includes a receiver and a transmitter. The transmitter and receiver belong to the physical layer of the SATA interface. The transmitter of the host interface 5108 of the UFS host 5100 is connected to the receiver of the device interface 5201 of the UFS device 5200, while the transmitter of the device interface 5201 of the UFS device 5200 is connected to the receiver of the host interface 5108 of the UFS host 5100.

[0119] UFS device 5200 can be connected to UFS host 5100 via device interface 5201. Host interface 5108 and device interface 5201 can be connected to each other via data lines for exchanging data or signals and via power lines for providing power.

[0120] UFS device 5200 may include device controller 5202, buffer memory 5204, and non-volatile storage device 5206. Device controller 5202 can control the overall operation of write, read, and erase operations such as those performed on non-volatile storage device 5206. Device controller 5202 can exchange data with buffer memory 5204 or non-volatile storage device 5206 via address and data buses. Device controller 5202 may include a central processing unit (CPU), device direct memory access (DMA), flash DMA, command manager, buffer manager, flash translation layer (FTL), flash manager, etc.

[0121] UFS device 5200 provides commands received from UFS host 5100 to device DMA and command manager via device interface 5201. Command manager can allocate buffer memory 5204 to receive data through buffer manager and send response signal to UFS host 5100 when data transfer is ready.

[0122] UFS host 5100 can send data to UFS device 5200 in response to an acknowledgment signal. UFS device 5200 can store the sent data in buffer memory 5204 via device DMA and buffer manager. The data stored in buffer memory 5204 is provided to flash manager via flash DMA. Then, flash manager can store the data at a selected address on non-volatile storage device 5206 with reference to the address mapping information of flash translation layer (FTL).

[0123] When the data transfer and programming required by the command from the UFS host 5100 are completed, the UFS device 5200 can send a response signal to the UFS host 5100 through the device interface 5201 to notify that the command is complete. The UFS host 5100 can then notify the device driver 5104 and the application 5102 of the completion of the command associated with the received response signal and terminate the command.

[0124] The device controller 5202 in the UFS system 5000 may include a reference Figures 1 to 11 The storage controller 100 is described. According to some embodiments, the device controller 5202 can perform operations related to... Figure 2 The processing unit 120 performs reinforcement learning-related operations. For example, the device controller 5202 can perform the same functions as the storage controller 100. According to some embodiments, the device controller 5202 can perform reinforcement learning-related operations in the learning processing unit 5400, which is related to... Figure 10 or Figure 11 The learning processing unit 170 is the same.

[0125] By enhancing learning, the size of a single-level cell region is adjusted based on the environmental information of the UFS system 5000, and dynamically adapted to the current state of the UFS system 5000. Environmental information includes, for example, pattern information of read, write, or delete operations of the UFS host 5100, and internal or external state information of the UFS device 5200.

[0126] In some embodiments, the non-volatile storage device 5206 may include a single-level cell region and a multi-level cell region 5250.

[0127] According to some embodiments, the buffer memory 5204 may include Figure 2 Register 140. According to some embodiments, buffer memory 5204 may include... Figure 2 The memory 130.

[0128] Figure 15 This is a block diagram illustrating an example embodiment of a solid-state storage device (e.g., an SSD) according to a concept of the present invention.

[0129] refer to Figure 15 The SSD system 6000 includes a host 6100 and an SSD 6200. The SSD 6200 exchanges signals SIG with the host 6100 via a signal connector and receives power PWR via a power connector. The SSD 6200 includes an SSD controller 6210, multiple non-volatile memory (NVM) units 6230, 6240, and 6250, and an auxiliary power supply 6220.

[0130] The SSD controller 6210 can control multiple non-volatile memories 6230, 6240, and 6250 in response to a signal SIG received from the host 6100. As an example, the SSD controller 6210 and the non-volatile memories 6230, 6240, and 6250 may include reference... Figures 1 to 12 The memory controller 100 and the non-volatile memory 200 are described.

[0131] At least one of the plurality of non-volatile memories 6230, 6240 and 6250 may include a single-level cell region and a multi-level cell region 6235.

[0132] According to an example embodiment, the SSD controller 6210 includes a learning processing unit 6300. The learning processing unit 6300 can perform reinforcement learning on the size of a single-level cell region based on environmental information of the SSD system 6000. In the example embodiment, the learning processing unit 6300 performs reinforcement learning whenever the size of write data received from the host 6100 is equal to or greater than the size of the single-level cell region.

[0133] According to some embodiments, the SSD controller 6210 can replace a separate learning processing unit to perform [the task]. Figure 2 Reinforcement learning of the processing unit 120 shown.

[0134] Auxiliary power supply 6220 is connected to host 6100 via a power connector. Auxiliary power supply 6220 can receive and charge power (PWR) from host 6100. When power from host 6100 is unstable or interrupted, auxiliary power supply 6220 can provide power to SSD system 6000. As an example, auxiliary power supply 6220 can be located inside or outside SSD 6200. For instance, auxiliary power supply 6220 can be located on the motherboard of SSD system 6000 and provide auxiliary power to SSD 6200.

[0135] During a sudden power outage, the SSD 6200 can access the non-volatile memories 6230, 6240, and 6250 using the charging power from the auxiliary power supply 6220.

[0136] As described above, at least one exemplary embodiment of the present invention uses reinforcement learning operations that take into account the environmental factors of the system to dynamically adjust the relative sizes of the SLC region (e.g., cache) and MLC region of the memory device of the memory system.

[0137] In closing, those skilled in the art will understand that many variations and modifications can be made to these exemplary embodiments without substantially departing from the principles of the inventive concept.

Claims

1.A storage device, comprising: at least one non-volatile memory including a single-level cell region and a multi-level cell region; and a storage controller configured to dynamically resize the single-level cell region by reinforcement learning, wherein the reinforcement learning uses environmental information of the storage device to determine a selected state among a plurality of states, calculates a current reward from the selected state and the environmental information, determines a reward direction based on the current reward and a reward history associated with the selected state, and adjusts the size of the single-level cell region according to the determined reward direction. 2.The storage device of claim 1, wherein the storage controller performs the reinforcement learning whenever a total amount of write data received from a host reaches a threshold bin count value. 3.The storage device of claim 2, wherein the storage controller includes a queue table including a plurality of entries, wherein each entry indicates the reward history and one of the plurality of states, and the storage controller adjusts the threshold bin count value according to the calculated reward direction. wherein 4.A method of operating a storage device, the method comprising: dividing a non-volatile memory into a single-level cell region and a multi-level cell region; performing reinforcement learning by using environmental information of the storage device to determine a selected state among a plurality of states, calculating a current reward from the selected state and the environmental information, determining a reward direction based on the current reward and a reward history associated with the selected state; and dynamically adjusting a size of the single-level cell region according to the determined reward direction. 5.The method of claim 4, further comprising: adjusting a threshold bin count value according to the determined reward direction; and adjusting the size of the single-level cell region according to the adjusted threshold bin count value. 6.The method of claim 5, wherein performing the reinforcement learning comprises: collecting the environmental information from the non-volatile memory and a host during a current cycle. 7.The method of claim 6, wherein the environmental information includes space utilization information, valid page count information, invalid page count information, and valid / invalid page count ratio information about the non-volatile memory in the current cycle. 8.The method of claim 6, wherein the environmental information includes an average write bin count value of the host. 9.The method of claim 6, wherein the environmental information includes history information about adjustments of the size of the single-level cell region in previous cycles. 10.A storage device, comprising: a non-volatile memory including a single-level cell region and a multi-level cell region; and a storage controller including a processing unit configured to perform reinforcement learning on environmental information of the storage device to determine an optimal ratio of the single-level cell region to the multi-level cell region, and wherein the storage controller dynamically adjusts a size of the single-level cell region according to the determined optimal ratio. ​ ​ ​ ​ The storage controller includes a memory storing a queue list storing a plurality of states respectively mapped to a plurality of pieces of environment information and a reward history corresponding to each of the plurality of states. 11.The storage device of claim 10, wherein the processing unit performs the reinforcement learning when predicted write data of the host to be written to the single-level cell region completely fills the single-level cell region. 12.The storage device of claim 11, wherein the processing unit resizes the single-level cell region at a current state and collects the environment information of the storage device during a current period. 13.The storage device of claim 12, wherein the storage controller includes a register configured to store the size of the single-level cell region and the environment information at the current state. 14.The storage device of claim 12, wherein the processing unit determines the current state of the non-volatile memory based on the environment information. 15.The storage device of claim 14, wherein the processing unit determines a reward from the determined current state and the environment information. 16.The storage device of claim 15, wherein the reward is determined based on a migration cost in a previous period and an effective / ineffective page count ratio in the current period. 17.The storage device of claim 16, wherein the reward is determined as a negative reward for resizing the single-level cell region when the migration cost of the previous period is greater than or equal to an average migration cost, when the effective / ineffective page count ratio of the single-level cell region in the current period is less than the effective / ineffective page count ratio of the single-level cell region in the previous period, and as a positive reward when the effective / ineffective page count ratio of the single-level cell region in the current period is not less than the effective / ineffective page count ratio of the single-level cell region in the previous period. 18.The storage device of claim 16, wherein the reward is determined as a negative reward for resizing the single-level cell region when the migration cost of a previous period is less than an average migration cost. 19.The storage device of claim 18, wherein the negative reward for resizing the single-level cell region increases when the migration cost of the previous period is less than the average migration cost, when the effective / ineffective page count ratio of the multi-level cell region in the current period is less than the effective / ineffective page count ratio of the multi-level cell region in the previous period.

Images