Storage controller and storage device including storage controller and method of operation thereof

Abstract

This disclosure provides a storage controller and a storage device including the storage controller, and a method for operating the same. The method includes: receiving user data and environmental information, obtaining logical characteristic information and physical characteristic information, defining a current state, obtaining a desired value, and performing a write operation. User data is received from a host device, and environmental information is received from non-volatile memory. The current state can be defined based on the logical characteristic information and physical characteristic information. The desired value can be obtained based on policy information and the current state. The write operation can be performed on the user data through a physical stream corresponding to the maximum value in the desired value.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0008754, filed on January 22, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0002] The inventive concept relates to a storage controller, and more specifically, to a storage device including a storage controller and a method of operating the storage controller. Background Technology

[0003] Storage devices are used to store information in a computer. Hard disk drives, solid-state drives, and flash memory are examples of storage devices. Storage devices can be organized into storage arrays with multiple storage devices. A storage controller is used to manage the storage devices in the storage array, determining which storage device to write data to and from which storage device to read data.

[0004] Artificial neural networks (ANNs) are computational systems designed to mimic biological systems like neurons. Machine learning is the field of artificial intelligence related to the use of ANNs. Machine learning generates new information by feeding data into a computer and updating the parameters of the ANN based on that data. Deep learning refers to training an ANN with multiple networks or layers.

[0005] In some cases, storage controllers can perform stream classification to improve efficiency. However, traditional stream classification increases development complexity and is unsuitable for responding to a wide variety of internal conditions. This degrades the performance of the storage device. Therefore, there is a need in the art to implement ANN and deep learning methods into storage controllers to improve the performance of read and write operations. Summary of the Invention

[0006] The inventive concept provides a memory controller capable of increasing performance metrics such as write amplification factor (WAF), invalid page count (VPC), and throughput by performing multi-stream classification using reinforcement learning, a memory device including the memory controller, and a method of operating the memory controller.

[0007] According to one aspect of the inventive concept, a method for operating a storage controller is provided, the method comprising: receiving user data and environmental information of a non-volatile memory; obtaining logical characteristic information and physical characteristic information; defining a current state based on the logical characteristic information and physical characteristic information; obtaining an expected value based on policy information and the current state; and performing a write operation on the user data through a physical stream corresponding to the maximum value among the expected values.

[0008] According to another aspect of the inventive concept, a storage controller is provided, the storage controller comprising: a multi-stream manager configured to generate stream IDs; a feature information generator configured to receive user data and environmental information, and generate logical feature information and physical feature information; and a reinforcement learning (RL) agent configured to: perform control based on the logical feature information and physical feature information to define a current state, obtain an expected value for each physical stream based on the current state and policy information about the physical streams selectable in the current state, and perform a write operation on user data through the physical stream corresponding to the maximum value among the expected values.

[0009] According to another aspect of the inventive concept, a storage device is provided, the storage device including a first memory block, a second memory block, and a storage controller, the storage controller being configured to send data to one of the first and second memory blocks according to a stream ID, wherein the storage controller is further configured to: receive user data from a host device; receive environmental information from each of the first and second memory blocks; generate logical characteristic information and physical characteristic information; define a current state based on the logical characteristic information and physical characteristic information; obtain an expected value regarding the selection of the first and second memory blocks based on the current state and policy information; and perform a write operation on the user data through the memory block corresponding to the maximum value among the expected values.

[0010] According to another aspect of the inventive concept, a method for data storage includes: receiving data to be stored in a memory; identifying a current state based on logical characteristic information about the data and physical characteristic information about the internal environment of the memory; classifying the data using a reinforcement learning agent trained to maximize the performance of the memory; and storing the data in the memory based on the classification of the data. Attached Figure Description

[0011] Embodiments of the present disclosure of the inventive concept will become clearer from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 This is a diagram illustrating an example of logical flow;

[0013] Figure 2 This is a block diagram of a storage system according to an embodiment of the present disclosure;

[0014] Figure 3A This is a block diagram of a storage controller according to an embodiment of the present disclosure;

[0015] Figure 3B This is a block diagram of a reinforcement learning (RL) agent according to embodiments of the present disclosure;

[0016] Figure 4 An example of a state function according to an embodiment of the present disclosure is shown;

[0017] Figure 5 An example of state function progression according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A method of operating a storage controller according to an embodiment of the present disclosure is illustrated;

[0019] Figure 7 An operational method of an RL agent according to an embodiment of the present disclosure is illustrated;

[0020] Figure 8 An example of repeated execution of RL according to embodiments of the present disclosure is shown; and

[0021] Figure 9 Another example of an update strategy according to an embodiment of this disclosure is shown. Detailed Implementation

[0022] This disclosure relates to a storage controller. More specifically, the disclosure relates to a storage device including a storage controller and a method of operating the storage controller. The disclosed embodiments use reinforcement learning applied to train a neural network for classifying streams of input data to the storage device. In one example, both the characteristics of the input data and the characteristics of the physical stream are defined as states, actions regarding which physical stream to select are defined, and a target performance metric is defined as a reward.

[0023] Reinforcement learning is one of the three fundamental machine learning paradigms that coexist with supervised and unsupervised learning. Specifically, reinforcement learning involves how a software agent makes decisions to maximize rewards. The decision-making model can be called a policy. This type of learning differs from supervised learning in that it does not require labeled training data and does not require explicit error correction. Instead, reinforcement learning balances the exploration of unknown options with the utilization of existing knowledge.

[0024] In some cases, reinforcement learning algorithms utilize dynamic programming techniques. However, one difference between reinforcement learning and other dynamic programming methods is that reinforcement learning does not require an exact mathematical model. Therefore, reinforcement learning models can be used for large models where exact methods are impractical.

[0025] In the context of storage devices, reinforcement learning models can be used to classify physical flows that enable the storage controller to achieve target performance metrics. The classification of various states can be selected by accumulating training datasets through multiple training processes and updating the policy based on the training datasets. As a result, the performance of the storage device can be improved.

[0026] In the following, embodiments of the inventive concept will be described in detail with reference to the accompanying drawings.

[0027] Figure 1 This is a diagram illustrating an example of logical flow.

[0028] A logical stream can refer to a set of data sent from a host device connected to a storage device. According to various embodiments of this disclosure, a logical stream can be referred to as a data stream. Criteria used to distinguish this set of data may include, at a minimum, logical block address (LBA) and throughput.

[0029] Reference Figure 1 Figure (a) shows the LBA values ​​of data input from the host device over time. Referring to Figure (a), each piece of data input continuously over time can constitute a logic stream. For example, each of the multiple segments shown in Figure (a) can correspond to a logic stream.

[0030] Reference Figure 1 Figure (b) illustrates the LBA values ​​of data input from the host device based on the processing speed. Referring to Figure (b), the data can be divided into two logical streams based on the processing speed and the LBA range. For example, the lower right logical stream may include data with a relatively high processing speed and a relatively small LBA value, while the upper left logical stream may include data with a low processing speed and a large LBA value.

[0031] When a storage device supports multi-stream, one of the streams included in the multi-stream can be referred to as a physical stream. (See reference...) Figure 1 If the storage device supports multiple streams and the number of physical streams is large, the number of logical streams can be even larger. In this case, a clustering problem may arise because multiple logical streams may be classified and assigned to a finite number of physical streams. For example, in the case of Figure (a), there may be a task of determining whether to assign data with short segment lengths (short-term transmissions) to the same physical stream or data with similar LBA ranges to the same physical stream.

[0032] Machine learning clustering techniques (such as K-Nearest Neighbor (K-NN) or K-means) can be applied to determine physical flows for allocating multiple logical flows. However, K-NN or K-means techniques may involve classifying data based on the similarity between sent data, and allocating data with similar characteristics to the same physical flow may not lead to performance improvements for desired indicators (such as write amplification factor (WAF) or effective page count (VPC)). Therefore, multiple logical flows can be clustered based on the characteristics of the real-time input data and the characteristics of the storage device, as will be discussed below. Figures 2 to 9 Describe it.

[0033] Figure 2 This is a block diagram of a storage system 1000 according to an embodiment of the present disclosure.

[0034] The storage system 1000 can be implemented as, for example, a personal computer (PC), a data server, a network attached storage (NAS), an Internet of Things (IoT) device, or a portable electronic device.

[0035] According to various embodiments of this disclosure, portable electronic devices may include laptop computers, mobile phones, smartphones, tablet PCs, personal digital assistants (PDAs), enterprise digital assistants (EDAs), digital still cameras, digital video cameras, audio devices, portable multimedia players (PMPs), personal navigation devices (PNDs), MP3 players, handheld game consoles, e-books, wearable devices, etc.

[0036] According to various embodiments of this disclosure, host device 20 can provide command CMD to storage device 10 and send and receive data DAT. For example, host device 20 can provide write command W_CMD and write data W_DAT to storage device 10. As another example, host device 20 can send read command R_CMD to storage device 10 and receive read data R_DAT from storage device 10.

[0037] According to various embodiments of this disclosure, host device 20 may include one or more processor cores. Additionally or alternatively, host device 20 may also be implemented as a system-on-a-chip. For example, host device 20 may include a general-purpose processor, a dedicated processor, or an application processor. Host device 20 may be the processor itself, or may correspond to an electronic device or system that includes a processor.

[0038] According to various embodiments of this disclosure, storage device 10 may include storage controller 100 and memory device 200. Memory device 200 may include a plurality of non-volatile memories 200_1 (i.e., NVM#1) to 200_N (i.e., NVM#N), where N is an integer greater than 1. Each of the plurality of non-volatile memories 200_1 to 200_N may include a memory region for storing data provided from host device 20. For example, each of the plurality of non-volatile memories 200_1 to 200_N may include an array of memory cells formed along a plurality of word lines and a plurality of bit lines.

[0039] In embodiments of this disclosure, the storage device 10 may be embedded in a storage system (e.g., Figure 2The storage device 10 is an embedded memory in the storage system 1000. For example, the storage device 10 may be an embedded multimedia card (eMMC) or an embedded universal flash memory (UFS) device. In embodiments of this disclosure, the storage device 10 may be an external memory removable from the storage system 1000. For example, the storage device 10 may be a UFS memory card, a compact flash memory (CF) card, a secure digital card (SD) card, a micro-secure digital card (Micro-SD) card, a mini-secure digital card (Mini-SD) card, an extreme digital card (xD), or a memory stick.

[0040] According to various embodiments of this disclosure, each of the plurality of non-volatile memories 200_1 to 200_N can employ various types of memory (such as NAND flash memory, magnetic RAM (MRAM), spin-transfer torque MRAM, conductive bridged RAM (CBRAM), ferroelectric RAM (FeRAM), phase RAM (PRAM), resistive RAM, nanotube RAM, polymer RAM (PoRAM), nanofloating gate memory (NFGM), holographic memory, molecular electronic memory, insulator resistance change memory, etc.).

[0041] According to various embodiments of this disclosure, the storage controller 100 can generally control the storage device 10. According to embodiments of this disclosure, the storage controller 100 may include a multi-stream manager 105. The multi-stream manager 105 can classify the write data W_DAT sent from the host device 20 into multiple physical streams based on the characteristics of the write data W_DAT and environmental information of each of the multiple non-volatile memories 200_1 to 200_N. Referring later... Figures 3A to 9 A detailed description of the multi-stream manager 105.

[0042] Figure 3A This is a block diagram of a storage controller according to an embodiment of the present disclosure.

[0043] Reference Figure 3A The storage controller 100 may include a processor 110, a memory 120, a host interface 130, a memory interface 140, and a reinforcement learning (RL) intellectual property (IP) 150 (or hardware intellectual property unit). These components may communicate with each other via a bus 160, and the storage controller 100 may include other configurations in addition to those shown above.

[0044] According to various embodiments of this disclosure, processor 110 can control the overall operation of storage controller 100. For example, processor 110 can perform clustering on a logical stream received from host device 20 by executing RL agent 123.

[0045] Processor 110 may include a central processing unit or a microprocessor unit. In embodiments of this disclosure, processor 110 may be implemented as a multi-core processor (such as a dual-core processor or a quad-core processor).

[0046] Processor 110 can execute firmware for driving storage controller 100. The firmware can be loaded into memory 120 and executed. For example, processor 110 can perform management functions by executing the firmware for driving storage controller 100. Figure 1 Garbage collection of multiple non-volatile memories 200_1 to 200_N or flash translation layers used to perform address mapping, wear leveling, etc.

[0047] Memory 120 can operate under the control of processor 110. Memory 120 can be used as operating memory, cache memory, or buffer memory of processor 110. Software, firmware, and data for controlling memory controller 100 can be loaded into memory 120. Memory 120 can be implemented as a volatile memory such as dynamic random access memory (DRAM) or static random access memory (SRAM). Alternatively, memory 120 can be implemented as a resistive memory such as RRAM, PRAM, or MRAM.

[0048] According to various embodiments of this disclosure, host interface 130 can perform communication with host device 20. For example, host interface 130 can provide a physical connection between host device 20 and storage device 10. Host interface 130 can adjust the size of data exchanged with storage device 10 or convert the format of commands exchanged with storage device 10 in response to the transmission format (such as bus format) of host device 20. The bus format of host device 20 can be configured as at least one of Universal Serial Bus (USB), Small Computer System Interface (SCSI), Peripheral Component Interconnect (PCI) Fast, AT Attach (ATA), Parallel AT Attach (PATA), Serial AT Attach (SATA), and Serial Attach SCSI (SAS). The Non-Volatile Memory Fast (NVMe) protocol installed on host device 20 that exchanges data via PCI Fast can be applied to host interface 130.

[0049] According to various embodiments of this disclosure, the memory interface 140 can exchange data with the memory device 200. The memory interface 140 can write data to the memory device 200 via channel CH and read data from the memory device 200 via channel CH. For example, the memory interface 140 can send model information to the memory device 200 via channel CH and receive model information from the memory device 200 via channel CH.

[0050] Memory device 200 may include flash memory, which may include a 2D NAND memory array or a 3D (or vertical) NAND (VNAND) memory array. A 3D memory array is a circuit associated with the operation of an array or memory cell having active regions disposed on a silicon substrate, and is monolithically formed on or within the substrate at at least one physical level of the circuitry. The term "monolithic" means that each layer constituting the array is directly stacked over each lower layer of the array. In embodiments of this disclosure, the 3D memory array includes vertically arranged NAND strings such that at least one memory cell is positioned above another memory cell. At least one memory cell may include a charge trapping layer.

[0051] U.S. Patent Publications Nos. 7,679,133, 8,553,466, 8,654,587, and 8,559,235 and U.S. Patent Application Publication No. 2011 / 0233648 describe appropriate configurations of 3D memory arrays, which include multiple levels and share word lines and / or bit lines between the multiple levels, and can be combined herein in the referenced documents.

[0052] According to various embodiments of the present disclosure, memory 120 may include feature information generator 121, RL agent 123, policy information storage device 125, and replay memory 127.

[0053] According to various embodiments of this disclosure, the characteristic information generator 121 can obtain characteristic information of a logical flow based on user data sent from the host device 20. Hereinafter, the characteristic information of the logical flow will be referred to as logical characteristic information.

[0054] Logical characteristic information can refer to information about the characteristics of a flow of data sent from host device 20. For example, logical characteristic information may include at least one of LBA range information, bandwidth information, and average page size information. LBA range information, bandwidth information, and average page size information are example values. Logical characteristic information is not limited to these. Logical characteristic information may include any information that can represent a data flow received from host device 20. For example, logical characteristic information may include information such as at least one of request size and interval time.

[0055] According to various embodiments of this disclosure, the characteristic information generator 121 can obtain physical flow information based on environmental information sent from a plurality of non-volatile memories 200_1 to 200_N. Hereinafter, the characteristic information of the physical flow will be referred to as physical characteristic information.

[0056] Physical characteristic information can refer to information about the internal environment of non-volatile memory. For example, physical characteristic information may include at least one of overwrite ratio information, effective page count (VPC) information, and LBA range information for written data. However, VPC information, overwrite ratio information, and LBA range information are example values, and physical characteristic information is not limited to these. Physical characteristic information may include any information that can represent the characteristics of data stored in non-volatile memory. For example, physical characteristic information may include at least one of the following: number of superblocks, Sequential Address Ratio (SAR) information, Terabyte Write (TBW) information, number of bad blocks, read / write request ratio, information indicating whether LBA ranges are aligned, and LBA range size information.

[0057] According to various embodiments of this disclosure, the RL agent 123 can generate or train a neural network, learn a policy based on a training dataset, and receive expected values ​​of received input data based on the learned policy. For example, the RL agent 123 can receive information about the current state, and this information may include physical characteristic information and / or logical characteristic information. The RL agent 123 can output multiple expected values ​​corresponding to the current state based on the policy information. Therefore, the current state can correspond to the environment, and the multiple expected values ​​can correspond to the decisions of the RL agent 123.

[0058] In various embodiments of this disclosure, RL can refer to a machine learning method that learns the action to be performed in the current state. When the action is performed, a reward is provided, and learning can progress to maximize the reward. In some cases, the model is trained to maximize the cumulative reward. In RL, even if the reward value is small, the model can learn to select actions that increase the cumulative reward value over time. In some cases, future rewards can be modified based on a discount rate obtained by converting future rewards into present values.

[0059] Some implementations use a form of learning, or reinforcement learning, called Q-learning. Q-learning is a model-free reinforcement learning algorithm that learns a policy to determine the action an agent should take given the current state of its environment. Q-learning does not require a model of the environment and can handle problems involving stochastic transitions and rewards. In some cases, Q-learning is based on a Finite Markov Decision Process (FMDP) and finds the optimal policy by maximizing the expected value of the total reward at any and all successive steps starting from the current state. Q-learning can identify the optimal action selection policy for any given FMDP, a given infinite exploration time, and a partially stochastic policy. "Q" refers to the quality function based on the reward returned by the action taken, and is therefore used to provide reinforcement.

[0060] Additional or optional, refer to Figure 3B The RL inference unit 123_1 of the RL agent 123 can output multiple expected values, where each expected value corresponds to the current state based on the policy information stored in the policy information storage device 125. The number of expected values ​​can correspond to the number of physical flows in a multi-flow system. For example, when the storage device 10 supports two physical flows, the number of expected values ​​can be two.

[0061] In some embodiments, the RL agent 123 includes or contains information from an artificial neural network (ANN). An ANN is a hardware or software component comprising nodes (also known as artificial neurons) with connections that roughly correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (like a physical synapse in the brain). When a node receives a signal, it processes it and then sends the processed signal to other connected nodes. In some cases, the signals between nodes consist of real numbers, and the output of each node is computed as a function of the sum of its inputs. Each node and edge is associated with one or more node weights that determine how the signal is processed and sent.

[0062] During training, these weights are adjusted to improve the accuracy of the results (i.e., by maximizing the reward function). The weights of the edges increase or decrease the strength of the signal transmitted between nodes. In some cases, nodes have a threshold below which no signal is transmitted. In some examples, nodes are aggregated into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal passes through certain layers multiple times.

[0063] ANNs can include various types of models, including Convolutional Neural Networks (CNNs), Region Convolutional Neural Networks with Convolutional Neural Networks (R-CNN), Region Proposal Networks (RPNs), Recurrent Neural Networks (RNNs), Stacked Deep Neural Networks (DNNs), State-Space Dynamic Neural Networks (S-SDNNs), Deconvolutional Networks, Deep Belief Networks (DBNs), Restricted Boltzmann Machines (RBMs), Fully Convolutional Networks, Long Short-Term Memory (LSTM) Networks, Classification Networks, Deep Q-Networks (DQNs), Dual DQNs, Dueling DQNs, Distributed Reinforcement Learning, Multi-Step Learning, Experience Priority Replay (PER), Noisy DQNs, Classification DQNs, Rainbow DQNs such as GoogleNet, AlexNet, and VGG networks, but are not limited to these.

[0064] Some Q-learning systems use deep convolutional neural networks (CNNs) with layers that tile convolutional filters to simulate the effect of the receptive field. In some cases, reinforcement learning can be unstable or divergent when Q is represented using a nonlinear function approximator such as a neural network. This instability is based on the correlations present in the sequence of observations. For example, a small update to Q can significantly alter the policy, the data distribution, and the correlation between Q and the target value. Therefore, Deep Q-learning (DQN) techniques can leverage experience replay, a biologically motivated mechanism that continues using random samples of prioritized actions instead of recent ones. This reduces the correlations in the sequence of observations and smooths changes in the data distribution. Iterative updates adjust Q towards the target value, which is only periodically updated, further reducing the correlation with the target.

[0065] In some examples, the DQN model also utilizes a target network to fix the parameters of the objective function. In some examples, reward limiting techniques are used to replace all positive rewards with the same value and all negative rewards with different values. In some examples, frame skipping techniques are used to calculate the Q-value at periodic intervals to reduce computational cost.

[0066] The strategy information storage device 125 can store strategy information generated by the RL agent 123. Strategy information is used to select the action that maximizes the reward from among multiple actions corresponding to the current state. For example, the strategy information may include a quality function (i.e., the Q function).

[0067] According to various embodiments of this disclosure, the strategy information can be updated. (See also...) Figure 3B The RL learning unit 123_2 can learn a policy based on the training dataset, and the newly learned policy can be transmitted to the policy information storage device 125.

[0068] According to various embodiments of this disclosure, the RL learning unit 123_2 may use a replay memory 127 to prevent confirmation bias. The replay memory 127 can reduce the correlation between training data input to the RL learning unit 123_2 by uniformly random sampling from the training dataset. The replay memory may be implemented as one of a ring buffer, a circular buffer, and a circular memory buffer.

[0069] According to various embodiments of this disclosure, RL IP 150 may include one or more processors to accelerate operations performed by a model of a neural network. Additionally, RL IP 150 may include a separate memory storing programs corresponding to the model of the neural network. For example, the processor may correspond to a neural network processing unit (NPU) and may include a fixed-function engine for performing convolutional layers and a programmable layer for performing non-convolutional layers. As another example, the processor may be implemented as at least one of a graphics processing unit (GPU) for high-speed parallel processing and an application-specific integrated circuit (ASIC)-based tensor processing unit (TPU) for parallel processing of vector and matrix operations.

[0070] According to various embodiments of this disclosure, RL IP 150 may be referred to by various terms including neural network processing device, neural network integrated circuit, neuromorphic computing unit or deep learning device.

[0071] Figure 4 An example of a state function according to an embodiment of the present disclosure is shown. Hereinafter, for ease of description, the case where the number of streams supported by the storage device 10 is two will be described. However, embodiments of the present disclosure with inventive concept are not limited thereto, and it will be apparent that embodiments of the present disclosure can be applied to storage devices 10 supporting at least three streams.

[0072] Reference Figure 4 The state function can include logical characteristic information and physical characteristic information.

[0073] The characteristic information corresponding to the logical flow may include average page size, bandwidth, and LBA range #0. The physical characteristic information may include first physical characteristic information and second physical characteristic information. The first physical characteristic information may refer to the information of the first memory block 200_1 corresponding to the first physical flow (physical flow #1). The second physical characteristic information may refer to the information of the second memory block 200_2 corresponding to the second physical flow (physical flow #2). The first physical characteristic information may include the LBA range #1, VPC (i.e., effective page count #1), and overwrite ratio information (i.e., overwrite ratio #1) corresponding to the first memory block 200_1. The second physical characteristic information may also include the corresponding LBA range #2, VPC (i.e., effective page count #2), and overwrite ratio information (i.e., overwrite ratio #2) corresponding to the second memory block 200_2.

[0074] According to embodiments of this disclosure, Figure 3AThe processor 110 or the characteristic information generator 121 can send a monitoring command M_CMD to a plurality of non-volatile memories 200_1 to 200_N. The monitoring command M_CMD can be a command requesting environmental information for each of the plurality of non-volatile memories 200_1 to 200_N. Each of the plurality of non-volatile memories 200_1 to 200_N can send environmental information to the storage controller 100 in response to receiving the monitoring command M_CMD. The environmental information may include at least one of overwrite ratio information, VPC, LBA range of write data W_DAT, read / write request ratio, SAR, and number of superblocks. The characteristic information generator 121 can generate physical characteristic information about each of the plurality of non-volatile memories 200_1 to 200_N based on the received environmental information.

[0075] Reference Figure 4 Physical characteristic information includes, but is not limited to, three pieces of information such as LBA range, VPC, and overwrite ratio. Characteristic information generator 121 can use information included in the environmental information as physical characteristic information.

[0076] According to embodiments of this disclosure, the processor 110 and / or the feature information generator 121 may not send the monitoring command M_CMD to the plurality of non-volatile memories 200_1 to 200_N, but may instead send the monitoring command M_CMD to at least some of the plurality of non-volatile memories 200_1 to 200_N. For example, when it has been determined in a past state that data has been sent via the first physical stream, the internal environment of the second memory block 200_2 corresponding to the second physical stream may remain unchanged. Therefore, the processor 110 or the feature information generator 121 may send the monitoring command M_CMD to the first memory block 200_1.

[0077] Reference Figure 4 The logical characteristic information includes three pieces of information: average page size, bandwidth, and LBA range information, but is not limited to these. In addition to these three pieces of information, the characteristic information generator 121 can also use request size information and interval time information as logical characteristic information.

[0078] According to embodiments of this disclosure, the feature information generator 121 can bypass the generation of logical feature information and receive logical feature information from the host device 20. For example, the host device 20 can drive software for a separate predictive model for transmitting data to the storage device 10. The predictive model can extract information such as average page size, bandwidth, LBA range, and interval time based on the data stream transmitted to the storage device 10. Additionally or alternatively, the predictive model can transmit information such as average page size, bandwidth, LBA range, and interval time to the processor 110 of the storage controller 100 or the feature information generator 121.

[0079] The characteristic information generator 121 can receive logical characteristic information, first physical characteristic information, and second physical characteristic information, and configure a state function corresponding to the current state. The state function can be transmitted to the RL learning unit 123_2 or the RL inference unit 123_1 of the RL agent 123.

[0080] Figure 5 An example of a state function series according to an embodiment of the present disclosure is shown.

[0081] Reference Figure 5 The state S is shown. t State S t+1 and state S t+2 .

[0082] In state S t In this configuration, RL agent 123 can select a first physical stream, and the first memory block 200_1 can have a current overwrite ratio of 0.7 and a VPC of 150,000. The LBA range for storing write data is 0x00000000 to 0x10000000. Additionally or optionally, the second memory block 200_2 can have a current overwrite ratio of 0.5, a VPC of 200,000, and an LBA range for storing write data of 0x20000000 to 0x30000000.

[0083] The following text will describe state S. t State S t+1 and state S t+2 It is included in the training section.

[0084] RL agent 123 can randomly determine actions. These actions can refer to assigning a given logical flow to any physical flow.

[0085] As mentioned above, because state S t It is included in the training part, so state S t This could be a time when policy information is absent or not generated. Therefore, RL agent 123 can determine the action randomly. For example, RL agent 123 can select a physical flow based on whether it is close to the LBA range. Since the LBA range of input data is 0x10000000 to 0x12000000, a continuous LBA range can be achieved when input data sent from host device 20 is written to the first memory block 200_1. RL agent 123 can send action information indicating that the first physical flow has been selected to processor 110. Processor 110 can control the addition of a flow identifier (ID) or tag representing the first physical flow to the status S. t The corresponding logical flow.

[0086] According to various embodiments of the present invention, a reward corresponding to each state can be defined. The reward can represent a target performance metric that can change due to action choices in any state. For example, the reward may include at least one of WAF, VPC changes, latency values, and bandwidth values.

[0087] According to various embodiments of this disclosure, the reward can have a positive or negative value. Whether the reward is positive or negative can be determined based on a target performance metric. For example, a larger WAF value indicates a greater amount of additional writes caused by garbage collection. This can represent a decrease in the performance of storage device 10. When the WAF value is close to 1, it can represent an improvement in the performance of storage device 10 because storage device 10 does not need to perform garbage collection. Therefore, when the target performance metric is WAF, the size of the reward can increase as the WAF value approaches 1. When the WAF value becomes greater than 1, the size of the reward can decrease or may be negative.

[0088] As another example, when the target performance metric is a VPC, the reward size can decrease or even become negative as the number of VPCs increases. This is because as the number of VPCs increases, the amount of additional writes increases during garbage collection. Therefore, when the target performance metric is a VPC, the reward size can be inversely proportional to the number of VPCs.

[0089] Reference Figure 5 The physical characteristics information and WAF value of the first memory block 200_1 can be obtained through state S. t The next step is to assign the logic flow to action A, which is in the first physical flow. t The overwrite ratio can be increased from 0.7 to 0.9, the VPC can be increased by 10000, and the LBA range of the data written to the first memory block 200_1 can be changed from the LBA range 0x00000000 to 0x11000000 to the LBA range 0x00000000 to 0x12000000. This indicates a return value R. t The WAF value can be changed to 1.1.

[0090] In state S t+1 In this context, RL agent 123 can select a second physical flow. As mentioned above, because state S t+1 It is also included in the training part, so it is in state S t+1 The RL agent 123 does not determine actions based on policy information, but can randomly select physical flows to generate the training dataset. For example, the RL agent 123 can select a second physical flow with different LBA ranges based on logical flows. The processor 110 can receive action information from the RL agent 123 and add the flow ID or label to the state S. tThe corresponding logical flow. The flow ID or tag information may include information representing the second physical flow.

[0091] Reference Figure 5 The overwrite ratio of the second memory block 200_2 can be reduced to 0.385. This is because it includes memory blocks in state S. t+1 The probability that the data in the logical flow is unrelated to the data written to the second memory block 200_2 is high. Additionally or alternatively, the VPC can be increased by 60000 when new data is written. Based on the overwrite ratio and changes in the VPC, state S... t+1 Action A t+1 The return R t+1 It can be reduced. This is because the WAF value increases from 1.1 to 1.3.

[0092] RL agent 123 can be controlled by inputting state S t and state S t+1 The characteristic information of each element is used as training data to perform learning, the action information of the first physical flow and the second physical flow are selected, and the information related to the change of the WAF value is reported back to the RL learning unit 123_2. When a sufficient amount of training data is input, the RL agent 123 can output the expected value of the action corresponding to the current state through the RL inference unit 123_1.

[0093] In the above embodiments of this disclosure, the standard for the return corresponding to the target performance metric is WAF, but it is not limited thereto.

[0094] According to various embodiments of this disclosure, the reward can be based on scores obtained by measuring multiple performance metrics. The reward can be based on changes in WAF and VPC. In this case, the weights between WAF and VPC can differ. For example, as a criterion for evaluating the performance of storage device 10, the weights can be adjusted to reflect 60% changes in VPC and 40% changes in WAF. Additionally or alternatively, the weights can be variable to be changed according to the strategies of the users and operators of storage device 10.

[0095] Figure 6 A method of operating a storage controller 100 according to an embodiment of the present disclosure is shown.

[0096] Reference Figure 6In operation S610, the storage controller 100 can receive user data and environmental information. User data may refer to a data stream sent from the host device 20 along with the write command W_CMD. Environmental information may correspond to information representing the internal state of each of the plurality of non-volatile memories 200_1 to 200_N. In operation S620, the storage controller 100 can obtain logical characteristic information and physical characteristic information. Logical characteristic information may include information about logical flows, and physical characteristic information may include information about the internal state of the plurality of non-volatile memories 200_1 to 200_N. Logical characteristic information is characteristic information of the user data and may include at least one of LBA range, request size, interval time, average size, and bandwidth. Physical characteristic information may include at least one of overwrite ratio, VPC, LBA range, SAR, number of superblocks, and number of bad blocks.

[0097] The storage controller 100 can obtain logical and physical characteristic information by using the characteristic information generator 121. According to embodiments of this disclosure, the processor 110 can, in response to a request from the characteristic information generator 121, send a monitoring command M_CMD requesting environmental information to at least one of the plurality of non-volatile memories 200_1 to 200_N.

[0098] In operation S630, the storage controller 100 can define the current state based on the obtained physical and logical characteristic information. For example, the current state can be determined by a combination of the internal states of multiple non-volatile memories 200_1 to 200_N and the characteristics of the data to be written by the user.

[0099] In operation S640, the storage controller 100 can obtain the desired value based on policy information and the current state. The RL inference unit 123_1 of the storage controller 100 can output the desired value of each of the actions selectable in the current state based on the policy information and the current state.

[0100] For example, when storage device 10 supports four physical streams, RL inference unit 123_1 can output a first expected value to a fourth expected value. The first expected value to the fourth expected value can correspond to the four physical streams respectively. For example, processor 110 or RL agent 123 can select which physical stream to allocate and store the write data W_DAT in the current state to increase the target performance metrics.

[0101] In operation S650, the storage controller 100 can write user data via a physical stream corresponding to the maximum value among the obtained expected values. The processor 110 or RL agent 123 can identify the physical stream corresponding to the maximum value among the obtained expected values. The processor 110 or RL agent 123 can add tag information or stream ID information representing the identified physical stream to the logical stream and execute a write command W_CMD on the logical stream.

[0102] Figure 7 The operation method of RL agent 123 according to an embodiment of the present disclosure is shown.

[0103] Reference Figure 7 In operation S710, the RL agent 123 can perform training based on an initial state. For example, the initial state can correspond to the state when the storage device 10 is first used. Therefore, the characteristic information of the multiple non-volatile memories 200_1 to 200_N can correspond to 0. The RL agent 123 can repeat random action selection within a predefined time period. The segment length of the predefined time period can be set to be sufficient to generate sample data for learning. For example, the segment length of the predefined time period can be determined based on at least one of the batch size, learning rate, size of replay memory 127, and size of sample data.

[0104] In operation S720, the RL agent 123 can perform learning based on the training dataset and generate and update policy information. The RL learning unit 123_2 can receive input from the training dataset and generate policy information for outputting the expected value of the input. According to various embodiments of this disclosure, the RL agent 123 can perform learning based on the training dataset and repeatedly perform the operation of updating policy information. By repeatedly updating the policy information, the policy information can effectively determine the action corresponding to the current state.

[0105] In operation S730, RL agent 123 can receive characteristic information about the current state. The characteristic information may include physical characteristic information of at least one of the multiple non-volatile memories 200_1 to 200_N and logical characteristic information of the user data. A detailed description of operation S730 is provided below. Figure 6 The detailed description of operation S620 is redundant, so it will be omitted.

[0106] In operation S740, RL agent 123 can select an action related to the current state based on policy information and characteristic information. An action can refer to selecting the action corresponding to the maximum expected value from among multiple actions selectable in the current state. A detailed description of operation S740 is provided below. Figure 6 The detailed description of the operation S650 is redundant, so it will be omitted.

[0107] Figure 8 An example of repeated execution of RL according to an embodiment of this disclosure is shown.

[0108] Reference Figure 8 The RL agent 123 can execute multiple episodes. During an episode, the RL agent 123 can repeatedly select actions starting from the initial state S0 within a predetermined time period. The predetermined time period can be set to a time sufficient to generate sample data for learning. For example, the segment length of the predetermined time period can be determined based on at least one of the batch size, learning rate, size of replay memory 127, and size of sample data.

[0109] According to various embodiments of this disclosure, the RL agent 123 can perform a first event. In performing the first event, since policy information for determining the action is unavailable, the RL agent 123 can select a random action and store the reward value. (See also...) Figure 8 This confirms that the WAF value increases further when the first event ends. An increased WAF value indicates the presence of numerous additional write operations during garbage collection, potentially degrading the target performance metrics. For example, the return value may be negative.

[0110] The RL agent 123 can perform a second event. When the second event occurs, policy information for determining the action in the current state may already exist. This policy information can be learned and generated by the RL learning unit 123_2 based on the training dataset generated during the first event. For example, when the second event occurs, the RL agent 123 can perform a selection based on the action and reward chosen in the first event. (See reference...) Figure 8 It can be confirmed that at the end of the second event, the WAF value increases compared to the initial state S0. However, it can be confirmed that the WAF value corresponding to the end time of the second event decreases compared to the WAF value corresponding to the end time of the first event. For example, RL agent 123 can decrease the WAF value and increase the reward by selecting actions based on the learned policy information, based on the action and reward of the first event.

[0111] The RL agent 123 can perform a third event. In the event of a third event, policy information for determining the action in the current state may already exist. This policy information can be learned and updated by the RL learning unit 123_2 based on the training dataset generated during the second event (excluding the first event).

[0112] According to various embodiments of this disclosure, policy information can be based on a search method such as an epsilon-greedy search method. For example, the search method may include randomly selecting actions based on a specific probability. As a result, local optima caused by bias can be prevented.

[0113] The term ε-greedy refers to a method in reinforcement learning that balances exploration and exploitation by randomly choosing between exploration and exploitation. ε refers to the probability of choosing to explore. In some cases, exploitation is used most of the time, while the opportunity to explore is small.

[0114] Reference Figure 8 As can be seen, in the first and second events, the RL agent 123 selects the first state S1 as the action in the initial state S0. For example, the RL agent 123 can select a physical flow with an LBA range adjacent to the LBA range of the logical flow. Furthermore, in the third event, the RL agent 123 can randomly select the first "state S1" as the action based on a specific probability. For example, the RL agent 123 can select a physical flow with an LBA range different from the LBA range of the logical flow. At the end of the third event, it can be confirmed that the WAF value has decreased from the WAF value of the initial state S0. For example, the RL agent 123 can reduce the WAF value and increase the target performance metric by selecting actions based on the learned policy information, based on the actions and rewards in each of the first and second events.

[0115] RL agent 123 can perform the nth event. Here, n can be a positive integer with a large value. For example, n can be 1000. For example, RL agent 123 can perform the nth event while selecting actions to increase the target performance metric by updating the policy information when executing the first to the (n-1)th events. The WAF value corresponding to the end time of the nth event can be reduced compared to the WAF value of the initial state S0.

[0116] Therefore, a data storage method may include: receiving data to be stored in a memory; identifying a current state based on logical characteristics of the data and physical characteristics of the memory's internal environment; classifying the data using a reinforcement learning agent trained to maximize the memory's performance; and storing the data in the memory based on the data classification.

[0117] Figure 9 Another example of an update strategy according to an embodiment of this disclosure is shown.

[0118] Reference Figure 9 The storage system 1000 may include a host device 20 and a storage device 10. For example, the storage system 1000 may be connected to... Figure 2 The storage system 1000 shown corresponds to this.

[0119] According to various embodiments of this disclosure, the storage system 1000 can send a training dataset to the cloud system 2000. The training dataset can be a representation of data generated during execution. Figure 8 The events shown are the action information selected for each step and the data on changes in the feedback based on the action information. For example, with Figure 8 The training dataset corresponding to the first event can include data on the changes in action information and reward for each of the four steps from the initial state S0 to the fourth state S4.

[0120] According to various embodiments of this disclosure, storage system 1000 can send training datasets to cloud system 2000 according to predefined time periods. For example, storage system 1000 can send training datasets at each time at the end of each event. See also Figure 8 Because storage system 1000 can send the training dataset at each endpoint from the first event to the nth event, the number of training datasets sent to cloud system 2000 can be n. As another example, storage system 1000 can send the training dataset to cloud system 2000 at each predefined time. In this case, the time when the training dataset is sent to cloud system 2000 may not coincide with the end time of the event.

[0121] According to various embodiments of this disclosure, the policy information generator 3000 can receive a training dataset from the cloud system 2000. The policy information generator 3000 may include... Figure 3B The configuration of the RL learning unit 123_2 shown is the same as or similar to that of the RL learning unit 123_2. For example, even when the storage system 1000 does not include the RL learning unit 123_2, the policy information generator 3000 can refer to the cloud computing IP used to update policy information. For example, the storage system 1000 that sends the training dataset to the cloud system 2000 may not include the RL learning unit 123_2. The storage system 1000 can use the RL inference unit 123_1 to perform stream classification for optimizing the target performance metric of the input user data without including the RL learning unit 123_2. As a result, the overall load on the storage system 1000 can be reduced.

[0122] The policy information generator 3000 can generate and update policy information based on the training dataset received from the cloud system 2000. (See reference...) Figure 8 The policy information generator 3000 can receive the training dataset corresponding to the first event and generate policy information based on the training dataset. The policy information generator 3000 can update the policy information when receiving the training dataset for the second to the (n-1)th events.

[0123] The policy information generator 3000 can send policy information to the storage system 1000 via the cloud system 2000. The policy information reflects the policies of a training dataset for multiple events and can be used to select actions that yield the maximum reward. The storage system 1000 can periodically receive updated policy information by simply sending the training dataset to the cloud system 2000, and the RL inference unit 123_1 can select actions based on the updated policy information.

[0124] Although the inventive concept has been specifically shown and described with reference to embodiments thereof, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the claims.

Claims

1. A method for operating a storage controller supporting multi-stream operations, the method comprising: Receive user data and environmental information from non-volatile memory; Logical characteristic information and physical characteristic information are obtained, wherein the physical characteristic information is obtained for each of the multiple physical flows; The current state is defined based on logical and physical characteristics. The expected value of each of the multiple physical flows is obtained based on policy information and the current state; Select the physical flow corresponding to the maximum value among the multiple physical flows and the desired value; and Write operations are performed on user data using the selected physical stream. The policy information includes information for selecting the physical flow that achieves the greatest performance improvement in the current state, and the policy information is generated and updated by a reinforcement learning unit based on a training dataset obtained by training a neural network. The state function includes logical and physical characteristic information. The action of selecting which physical flow to use is defined, and the target performance metric is defined as the reward. The logical characteristic information is obtained based on user data and includes bandwidth information and average page size information. The physical characteristic information is obtained based on environmental information and includes logical block address range information corresponding to the memory block, overwrite ratio information, and effective page count information. The rewards include changes in the effective page count. Among these, strategy information is used to select the action that maximizes the reward from among multiple actions corresponding to the current state.

2. The operating method according to claim 1, wherein, The logical characteristic information also includes at least one of the following: the range of logical block addresses of user data, request size, and interval time.

3. The operating method according to claim 1, wherein, The physical characteristic information also includes at least one of the following: superblock quantity information, sequential address ratio information, terabyte write information, bad block quantity information, read / write request ratio information, information indicating whether the logical block address range is aligned, and logical block address range size information.

4. The operating method according to claim 1, wherein, The neural network is based on at least one of deep Q-networks, dual deep Q-networks, and competing deep Q-networks.

5. The operating method according to claim 1, wherein, The expected value is determined based on the improvement of the target performance index, and The target performance metric corresponds to at least one of the following: effective page count, write amplification factor, latency, and bandwidth.

6. The operating method according to any one of claims 1 to 5, wherein, The steps for performing a write operation on user data include: Identify the physical flow corresponding to the maximum value; and Add a flow identifier representing the identified physical flow to the user data.

7. A storage controller, the storage controller comprising: The multi-stream manager is configured to generate stream identifiers; A characteristic information generator is configured to receive user data and environmental information, and to generate logical characteristic information and physical characteristic information, wherein the physical characteristic information is obtained for each of multiple physical flows; and The reinforcement learning agent is configured to: define the current state based on logical and physical characteristic information; obtain an expected value for each physical flow based on the current state and policy information regarding the physical flows selectable in the current state; select the physical flow corresponding to the maximum expected value from the plurality of physical flows; and perform a write operation on user data through the selected physical flow. The reinforcement learning agent includes a reinforcement learning unit, which is configured to generate and update policy information based on a training dataset obtained by training a neural network. The strategy information includes information for selecting the physical flow that will achieve the greatest performance improvement in the current state. The state function includes logical and physical characteristic information. The action of selecting which physical flow to use is defined, and the target performance metric is defined as the reward. The logical characteristic information is obtained based on user data and includes bandwidth information and average page size information. The physical characteristic information is obtained based on environmental information and includes logical block address range information corresponding to the memory block, overwrite ratio information, and effective page count information. The rewards include changes in the effective page count. Among these, strategy information is used to select the action that maximizes the reward from among multiple actions corresponding to the current state.

8. The storage controller according to claim 7, wherein, The logical characteristic information also includes at least one of the following: the range of logical block addresses of user data, request size, and interval time.

9. The storage controller according to claim 7, wherein, The physical characteristic information also includes at least one of the following: superblock quantity information, sequential address ratio information, terabyte write information, bad block quantity information, read / write request ratio information, information indicating whether the logical block address range is aligned, and logical block address range size information.

10. The storage controller according to claim 7, wherein, The neural network is based on at least one of deep Q-networks, dual deep Q-networks, and competing deep Q-networks.

11. The storage controller according to claim 7, wherein, The expected value is determined based on the improvement of the target performance metric, and The target performance metric corresponds to at least one of the following: effective page count, write amplification factor, latency, and bandwidth.

12. The storage controller according to any one of claims 7 to 11, wherein, The reinforcement learning agent is also configured to identify the physical flow corresponding to the maximum value, and The multi-stream manager is also configured to add a stream identifier representing the identified physical stream to the user data.

13. The memory controller according to any one of claims 7 to 11, wherein the memory controller further comprises a hardware intellectual property unit for operating the neural network. in, The hardware intellectual property unit is implemented as a neural processing unit or a neuromorphic computing unit.

14. The storage controller according to any one of claims 7 to 11, wherein the storage controller further comprises a policy information storage device for storing policy information. in, Policy information is generated and updated by reinforcement learning learning units, or received from cloud servers.

15. A storage device for supporting multiple streams, the storage device comprising: First memory block; Second memory block; as well as The storage controller is configured to send data to one of the first and second memory blocks based on the stream identifier. The storage controller is further configured to: receive user data from the host device; receive environmental information from each of the first and second memory blocks; generate logical characteristic information and physical characteristic information of the first and second memory blocks; define a current state based on the logical and physical characteristic information; obtain an expected value for the selection of the first and second memory blocks based on the current state and policy information; select the memory block corresponding to the maximum value among the expected values; and perform a write operation on the user data using the selected memory block. The storage controller includes a reinforcement learning unit configured to generate and update policy information based on a training dataset obtained by training a neural network. The strategy information includes information for selecting the physical flow that will achieve the greatest performance improvement in the current state. The state function includes logical and physical characteristic information. The action of selecting which physical flow to use is defined, and the target performance metric is defined as the reward. The logical characteristic information is obtained based on user data and includes bandwidth information and average page size information. The physical characteristic information is obtained based on environmental information and includes logical block address range information corresponding to the memory block, overwrite ratio information, and effective page count information. The rewards include changes in the effective page count. Among these, strategy information is used to select the action that maximizes the reward from among multiple actions corresponding to the current state.

16. The storage device according to claim 15, wherein, The logical characteristic information also includes at least one of the following: the range of logical block addresses of user data, request size, and interval time.

17. The storage device according to claim 15 or claim 16, wherein, The physical characteristic information also includes at least one of the following: superblock quantity information, sequential address ratio information, terabyte write information, bad block quantity information, read / write request ratio information, information indicating whether the logical block address range is aligned, and logical block address range size information.

18. The storage device according to claim 17, wherein, Environmental information is generated in response to the first and second memory blocks receiving monitoring commands and is sent to the storage controller.

Images