Machine learning for temperature compensation

Abstract

A method of performing temperature compensation to read a flash memory device includes determining a state of the flash memory device. An action having a maximum Q value from a Q table is selected for a current state during mining. A read operation of a codeword is performed from the flash memory device using one or more parameters according to the selected action. The codeword is decoded with an error correction code (ECC) process.

Description

BACKGROUND

[0001] The present disclosure relates generally to data storage devices, and more specifically to machine learning for temperature compensation (TC) to read a flash memory device.

[0002] Non-volatile data storage devices, such as flash memory devices, have enabled increased portability of data and software applications. Flash memory devices can enhance data storage density by storing multiple bits in each flash memory cell. For example, flash memory devices provide increased storage density by storing 2 bits per cell (i.e., MLC cells), 3 bits per cell (i.e., TLC cells), 4 bits per cell (i.e., QLC cells), or more. While increasing the number of bits per cell increases the storage density of a flash memory device, the bit error rate (BER) of data stored at the flash memory device can also increase.

[0003] Furthermore, increased BER due to temperature changes between program temperature (e.g., the temperature of a flash memory device when data is written to the flash memory device) and later read temperature (e.g., the temperature of the flash memory device when data is read from the flash memory device) is becoming an increasingly significant problem. Because each storage element can have a distinct cross-temperature coefficient, each storage element can exhibit different threshold voltage (Vt) shifts due to temperature changes relative to the temperature at which the storage element was programmed and verified. The Vt shift per storage element is a function of temperature difference. Thus, reading a flash memory device at a temperature different from the program temperature results in shifts and widening of the cell voltage distribution (CVD) for different states of the flash memory device, and increased BER. The shifts and widening of the CVD and increased BER result in two temperature change directions, such as when data is read from a storage element at a higher temperature than the data was written to the storage element, and when data is read from a storage element at a lower temperature than the data was written to the storage element.

[0004] One previous method for temperature compensation of a flash memory device includes searching over a range of flash memory device parameters, such as multiple simple reads with different default parameter values, followed by threshold calibration and bit error rate (BER) estimation. The search is stopped once the BER of a codeword is low enough to be decoded and successful decoding is achieved. However, this search can have a long latency. Thus, there is a need for an improved method of managing temperature compensation of a flash memory device. SUMMARY

[0005] In one embodiment, a method of performing temperature compensation to read a flash memory device includes determining a state of the flash memory device. An action having a maximum Q value from a Q-table is selected for the current state during mining. A read operation of a codeword from the flash memory device is performed using one or more parameters according to the selected action. The codeword is decoded with an error correction code (ECC) process.

[0006] In one embodiment, a data storage device includes a memory die. A circuit bounded array (CbA) is coupled to the memory die. The CbA includes Q-learning logic comprising a BER estimation module, a Q-table, and a Q-learning process configured to converge to an optimal policy for an agent and update Q-table values. The Q-learning logic is operable to determine a current state of the flash memory device; select an action having a maximum Q value from the Q-table for the current state during mining; and perform a read operation of a codeword from the memory die using one or more parameters according to the selected action.

[0007] In another embodiment, a data storage device includes a memory die. A controller is coupled to the memory die. The controller includes a BER estimation module; a Q-table; and an ECC engine. The controller is operable to determine a current state of the memory die; select an action having a maximum Q value from the Q-table for the current state during mining; perform a read operation of a codeword from the memory die using one or more parameters according to the selected action; and decode the codeword with the (ECC) engine. BRIEF DESCRIPTION OF DRAWINGS

[0008] So that the above-recited features of the present disclosure can be understood in detail, a more particular description will be rendered by reference to example embodiments, some of which are illustrated in the drawings. It is to be noted, however, that the appended drawings are designed solely for purposes of illustration and not as a definition of the limits of the disclosure, for the disclosure can admit to other equally effective embodiments.

[0009] Figure 1 is a schematic block diagram illustrating certain embodiments of a computer system including a host connected to a data storage device.

[0010] Figure 2 is a schematic illustration of a recursive reinforcement learning (RL) process for a TC that selects an action for a current environment state of an agent as a Markov Decision Process.

[0011] Figures 3A-3C is a schematic illustration of certain embodiments of Q-learning logic performing a Q-learning process for a TC to read a flash memory die.

[0012] Figure 4 is a schematic illustration of certain embodiments of a set of agent actions selected from a Q-table.

[0013] Figure 5A -B is a schematic flow diagram of an embodiment of a Q-learning process for TC to read a flash memory device.

[0014] Figure 6 is a schematic illustration of an instance of certain embodiments of a recursive Q-learning process for TC performed by a storage device controller.

[0015] Figure 7 is a schematic illustration of an instance of certain embodiments of a recursive Q-learning process for TC performed by Q-learning logic associated with a NAND flash memory die.

[0016] Figure 8 is a sigma plot of the failure bit count for reads of a NAND flash memory die in cross-temperature conditions under various searches.

[0017] For ease of understanding, the same reference numbers have been used in the drawings to designate the same elements. It is contemplated that elements disclosed in one embodiment can be advantageously utilized on other embodiments without specific recitation. DETAILED DESCRIPTION

[0018] In the following, reference is made to embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to the particular embodiments described. Indeed, any combination of the following features and elements, whether related to different embodiments or not, is contemplated to implement and practice the present disclosure. Additionally, no limitation of the scope of the disclosure is intended by the inclusion of the dependent claims, the presentation of a feature or element in the claims section, or the inclusion of a corresponding statement in the background. Furthermore, it will be understood that the use of "comprise", "comprises", "comprising", "containing", "contains", "contain" or variations thereof herein, is to specify the presence of stated features, elements, and / or components, but does not preclude the presence or addition of one or more other features, elements, and / or components. It is contemplated that the features and elements disclosed in the various embodiments can be interchanged or combined with each other in any manner.

[0019] Embodiments of data storage devices are operable to utilize machine learning (ML) for temperature compensation to read flash memory devices. The machine learning can be implemented in logic implemented on a circuit bounded array coupled to a flash memory die or on a storage device controller. The machine learning process determines an environmental state of a flash memory die and selects one of a plurality of read actions corresponding to the state. Each read action includes at least one TC parameter with a default threshold voltage or with threshold voltage compensation. A read of a codeword from the flash memory die is performed using the selected parameters and threshold voltage of the selected read action. If an estimated BER of the codeword is at or below a decode limit, the codeword is decoded by an error correction code (ECC) process. If the estimated BER of the codeword is above the decode limit, another read action is selected. The read actions are selected based on cumulative rewards of read actions that achieve successful decoding, targeting to reduce cumulative decode latency during multiple read operations to the flash memory device.

[0020] The selection of TC parameters is determined by a Markov decision process (MDP) with the goal of minimizing cumulative decode latency. The ML process can be a reinforcement learning (RL) process that solves the MDP by maximizing cumulative rewards. The RL process selects TC parameters for multiple environmental states of multiple read operations to the flash memory device with high accuracy and with low cumulative decode latency, through default threshold voltages and through threshold voltage calibration. In certain embodiments, RL with offline training helps to provide TC parameters during early life of the flash memory device. In certain aspects, RL with online training helps to adapt to changes and variations in the flash memory device during life of the flash memory device. The RL process can be performed by logic implemented on each flash memory device, e.g., on each NAND flash memory die, or can be implemented on a storage device controller.

[0021] Figure 1 is a schematic block diagram illustrating certain embodiments of a computer system 100 including a data storage device 102, such as a solid state drive (SSD), connected to a host 150. The host 150 utilizes non-volatile memory (NVM) 106, such as a flash memory device, included in the data storage device 102 to write and read data, e.g., for long term memory storage.

[0022] Host 150 can be a hardware platform that includes one or more central processing units (CPUs) 152, storage device interfaces 156, and host memory space 160. Host 150 can include a wide variety of devices, such as computer servers, network-attached storage (NAS) units, desktop computers, notebook (e.g., laptop) computers, tablet computers (e.g., "smart" pads), mobile devices, set-top boxes, telephone handsets (e.g., "smart" phones), televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, and automotive applications (e.g., mapping, autonomous driving). In certain embodiments, host 150 includes any device having a processing unit or any form of hardware capable of processing data, including general-purpose processing units, special-purpose hardware (e.g., application-specific integrated circuits (ASICs)), configurable hardware (e.g., field-programmable gate arrays (FPGAs)), or any other form of processing unit configured by software instructions, microcode, or firmware.

[0023] Host memory space 160 is a device that allows storage and retrieval of information, such as executable instructions, cryptographic keys, configurations, and other data. Host memory space 160 can be DRAM, SRAM, other main memory, and combinations thereof. Applications can be stored to host memory space 160 for execution by components of host 150.

[0024] Data storage device 102 can be a network storage device, an internal storage drive (e.g., a server computer, a desktop hard drive, a notebook computer), a data center storage device, an external storage device, an embedded mass storage device, a removable mass storage device, and other suitable data storage devices.

[0025] Host 150 can be connected to storage device 102 via wired links, wireless links, data center storage system networks, enterprise storage system networks, storage area networks, cloud storage networks, local area networks (LANs), wide area networks (WANs), the Internet, other networks, and interconnections thereof.

[0026] In certain embodiments, the storage device interface 156 can be a dedicated link between the host 150 and the data storage device 102. The storage device interface 156 of the host 150 interacts with the host interface 101 of the data storage device 102 for communication between the host 150 and the data storage device 102. In certain embodiments, the storage device interface 156 can be a network interface. The network interface enables the host 150 to communicate with the data storage device 102 via a communication medium such as a network coupling the host 150 and the data storage device 102 within the computer system 100. The network interface can be one or more network adapters, also known as network interface cards (NICs). The storage device interface 156 and the host interface 101 operate under a communication protocol such as a Peripheral Component Interface Express (PCIe) serial communication protocol, Universal Flash Storage (UFS), Serial Advanced Technology Attachment (SATA), Serial Attached SCSI (SAS), or other suitable communication protocol.

[0027] The data storage device 102 includes a storage device controller 110 that manages operations of the storage device 102, such as writes to and reads from the NVM 106. In certain embodiments, the storage device controller 110 includes an error correction code (ECC) engine 140 to decode data stored in the NVM 106. For example, the ECC engine 140 can use parity data to decode data using any suitable decoding algorithm. The RL process for TCs improves accuracy and / or reduces cumulative decoding latency by the ECC engine 140. For example, the RL process for TCs can reduce the number of iterations of an iterative decoding algorithm required by the ECC engine 140 to reach a valid codeword, and / or the RL process for TCs can enable use of a less powerful ECC engine 140.

[0028] The storage device controller 110 can include one or more processors 130, which can be multi-core processors. The processors 130 handle the components of the data storage device 102 via firmware and / or software. The processors 130 execute instructions of computer-readable program code of the firmware and / or software. The storage device controller 110 implements the processes described herein by execution of instructions by the processors 130, by hardware, or by a combination thereof. The hardware can include various components of the storage device controller 110, such as logic gates, switches, application specific integrated circuits (ASICs), programmable logic controllers, embedded microcontrollers, registers, and other components.

[0029] The NVM 106 of the storage device 102 can be configured for long-term storage of information as non-volatile memory space and to retain information after power on / off cycles. Data stored to the NVM 106 can also include metadata, such as write temperature and / or parity code associated with the stored data. The NVM 106 includes a plurality of NVM dies 108 of a flash memory device, such as NAND flash memory dies. The flash memory device includes arrays of single-level memory cells (SLC) storing one memory bit per cell, multi-level memory cells (MLC) storing two memory bits per cell, triple-level memory cells (TLC) storing three memory bits per cell, quad-level memory cells (QLC) storing four memory bits per cell, or other level memory cell technologies now known, in development, or later developed (e.g., five-level cell memory cells).

[0030] NAND flash memory devices are manufactured in two-dimensional or three-dimensional architectures. Temperature cross issues in NAND flash memory devices based on three-dimensional architectures are exacerbated due to the use of charge traps as compared to floating gates in NAND flash memory devices based on two-dimensional architectures. Each memory cell of a 3D NAND memory array can have its own different cross temperature coefficient and can exhibit different Vt shifts due to changes in program temperature and read temperature (i.e., cross temperature). As compared to 2D NAND memory arrays, 3D NAND memory arrays result in increased shifts and widening of CVD distributions of different states and increased BER due to cross temperature issues. In other embodiments, the NVM 106 can be other non-volatile memory, such as a NOR flash memory device.

[0031] The RL for TC can be implemented in a circuit-bounded array (CbA) 109 associated with each of the NVM dies 108, or on the storage device controller 110. The NVM 106 includes a temperature sensor 112. The temperature sensor 112 is configured to measure ambient temperature. For example, the temperature sensor 112 can provide a measured temperature, such as a read temperature, to the CbA 109 and / or the storage device controller 110. The NVM 106 can include a single temperature sensor 112 or can include multiple temperature sensors 112. For example, in a particular embodiment of the NVM 106, each of the NVM dies 108 can include a separate temperature sensor 112.

[0032] The embodiments described herein are not limited to the three-dimensional memory array described, but encompass all related memory structures as will be appreciated by one of skill in the art. The embodiments described herein are not limited to NAND flash memory devices, but encompass all related memory devices, such as NOR flash devices, ReRAM memory cells, or other memory arrays comprised of storage class memory. Multiple memory dies, such as memory arrays of the same type or mixed types, can be coupled together to form the NVM 106 of the data storage device 102.

[0033] Figure 2 is a schematic illustration of a recursive RL process 200 for the TC of the agent 210 picking an action as an MDP for a current environment state 220, where the goal is to minimize the cumulative decoding latency. RL solves MDP problems by maximizing the cumulative reward. In other embodiments, other ML, such as supervised learning or unsupervised learning, can be used to solve MDP problems. In certain embodiments, the agent 210 is Q-learning logic coupled to a NAND flash memory die, and the environment is a NAND flash memory die. In certain embodiments, the agent 210 is a storage device controller, and the environment is a NAND flash memory die.

[0034] The agent 210 selects an action based on the environment state 220 and maximizes the cumulative reward. The selected action affects the current reward and the future reward. The agent 210 takes the action, which in turn changes the environment state 220 and provides a reward. If the sum of the immediate reward plus the expected total reward of the next state is high, the agent 210 will learn to prefer more of the same action in this state. Otherwise, other actions can be picked in the future. The Q-learning process is configured to converge to the optimal policy of the agent and update the Q-table values accordingly. After converging to the optimal policy, the converged action that the agent selects in each state is the action with the maximum Q-value in the Q-table. Converging to the optimal policy can be achieved by offline training in the lab during production or in-field operation.

[0035] Further, the agent 210 uses an exploration value, such as an epsilon-greedy value 0 < ε < 1, to provide the agent 210 with a probability of exploring a new action versus exploiting the current action, for example, during the early life of a NAND flash memory die when the estimate of the expected total reward of the next state can not be accurate. For example, during exploration, the agent 210 selects an action using a probability p = 1 - ε.

[0036] The RL process 200 learns from experience of interacting with respective environmental states 220 of a memory die for multiple read operations to select respective actions that achieve successful decoding with low cumulative decoding latency. The RL process 200 uses rewards to select actions. Over time, actions that receive higher rewards will be preferred over actions with smaller rewards. In certain embodiments, the recursive RL process 200 finds and delivers TC parameters that achieve successful decoding with low cumulative decoding latency in combination with a default threshold voltage or a calibrated threshold voltage.

[0037] In certain embodiments, the RL process 200 is a Q-learning process with a quantized representation of states using a tiling and linear function approximation. For example, instead of representing a characteristic of an environmental state as a single number, the characteristic is represented as multiple values, alone or in combination with other characteristics. In certain aspects, Q-learning can provide a constraint that one or more characteristics correspond to an atomic action that is less time consuming. In certain aspects, Q-learning provides multiple actions that correspond to an environmental state that can be selected with negligible run time.

[0038] The goal of Q-learning is to learn an optimal policy that tells the agent 210 what action to take in what situation (e.g., state). Q-learning does not require a model of the environment. One example of a Q-learning recursive formula is shown in equation (I), although other Q-learning recursive formulas can be used:

[0039] Q new (S t ,A t )←Q current (S t ,A t )+∝[R t+1 +γmax Q(S t+1 ,a)-Q(S t ,A t )] (I)

[0040] The term Q current (S t ,A t ) is the Q-value for the current state-action pair. The term a is the learning rate. The term g is the discount factor. The term max Q(S t+1 ,a) is the maximum expected reward given the new state S t+1 and all possible actions for that state. The term Q new (S t ,A t ) is the updated Q-value for the current state-action pair.

[0041] The environmental state S t may include various characteristics of memory cells of a NAND flash memory die. In certain embodiments, the environmental state St one or more of the following characteristics: TC parameters of the current state, TC parameters of the previous state, time since programming, programming temperature, read temperature, number of program / erase cycles of a block of the NAND flash memory die, syndrome weight (SW) of the number of failed parity equations of the current state, syndrome weight of the previous state, decoding process of the ECC engine that achieved successful decoding, other suitable characteristics, and quantization thereof.

[0042] Table I shows an example of Q values of a Q-table, where an "n" number of states correspond to an "m" number of actions. The Q-table can include any number of states corresponding to any number of actions.

[0043]

[0044] The Q-table provides accumulated rewards for states and actions. The Q-table is updated by individual experiences of the NVM 106 of the data storage device 102 in the field used by the host 150. New experiences of the NVM 106 in the field further update the Q-table. After multiple iterations of Q-learning converge, each state has an action that has the maximum Q-value that achieves successful decoding with low accumulated decoding latency. In other embodiments, Q-learning utilizes function approximation, such as a neural network, rather than a Q-table. Q-learning using a Q-table or a neural network can be performed by logic implemented in each of the NVM dies 108 or on the storage device controller 110.

[0045] Figure 3A -C is a schematic illustration of certain embodiments of the recursive Q-learning process that the Q-learning logic 300 performs for TC to read the NAND flash memory dies. For ease of description, the Q-learning logic 300 is described with reference to the implementation in the data storage device 102 of Figure 1 The Q-learning logic 300 can be implemented in any suitable data storage device. In certain embodiments, the Q-learning logic 300 can run continuously (e.g., always on) during use of the data storage device 102 without triggering events. In certain embodiments, the Q-learning logic 300 can be triggered by a triggering event, such as a temperature difference between a read temperature and a program temperature of stored data being above a threshold, by a decoding failure of the ECC engine 140, by a request from the host 150, by a recovery mode, by a NAND die management mode, or by other triggering events.

[0046] In certain embodiments, the Q-learning logic 300 includes a threshold calibration module 310, a default threshold table 312, a BER estimation module 314, a temperature compensation parameter parameter change module 316, and a Q-table 320. In certain embodiments, the Q-table 320 includes a set of multiple actions using default threshold voltages and a set of multiple actions using threshold calibration.

[0047] The Q-learning agent determines a current environment state. Based on the current environment state, the Q-learning agent selects an action from a plurality of actions corresponding to the state. Typically, the action with the largest Q-value for the current environment state is selected. In the case where the goal is to minimize the cumulative decoding latency, the Q-learning agent selects a read action (e.g., a first read action) that has one or more TC parameters to read data using default threshold voltages or using calibrated threshold voltages. In embodiments where the Q-learning logic 300 runs continuously, the current environment state is the state of the flash memory device for sequential or random read operations for a previous read access of the NAND flash memory die 108. The selected read action from the Q-table 320 determines the next state.

[0048] In Figure 3A embodiments, the Q-learning logic 300 is implemented on a per-die basis. In such embodiments, the Q-learning logic 300 is implemented on a per-die basis for each of the NAND flash memory dies 108. For example, there are “N” number of Q-learning logic 300 for “N” number of NAND flash memory dies 108. One Q-learning logic 300 is coupled to one NAND flash memory die 108, as shown in FIG. 1. In such embodiments, the Q-learning logic 300 is implemented on a per-die basis for each of the NAND flash memory dies 108. Figure 3A In Figure 3A embodiments, the Q-learning logic 300 is implemented on a per-die basis. In such embodiments, the Q-learning logic 300 is implemented on a per-die basis for each of the NAND flash memory dies 108. For example, there are “N” number of Q-learning logic 300 for “N” number of NAND flash memory dies 108. One Q-learning logic 300 is coupled to one NAND flash memory die 108, as shown in FIG. 1. In such embodiments, the Q-learning logic 300 is implemented on a per-die basis for each of the NAND flash memory dies 108.

[0049] In Figure 3B embodiments, the Q-learning logic 300 is implemented on a per-die basis. In such embodiments, the Q-learning logic 300 is implemented on a per-die basis for each of the NAND flash memory dies 108. For example, there are “N” number of Q-learning logic 300 for “N” number of NAND flash memory dies 108. One Q-learning logic 300 is coupled to one NAND flash memory die 108, as shown in FIG. 1. In such embodiments, the Q-learning logic 300 is implemented on a per-die basis for each of the NAND flash memory dies 108. Figure 3B In embodiments, the Q-learning logic 300 is implemented on a per-die basis. In such embodiments, the Q-learning logic 300 is implemented on a per-die basis for each of the NAND flash memory dies 108. For example, there are “N” number of Q-learning logic 300 for “N” number of NAND flash memory dies 108. One Q-learning logic 300 is coupled to one NAND flash memory die 108, as shown in FIG. 1. In such embodiments, the Q-learning logic 300 is implemented on a per-die basis for each of the NAND flash memory dies 108.

[0050] In Figure 3CIn an embodiment of the present invention, the Q-learning logic 300 is implemented on an individual circuit bounded array (CbA) 309 coupled to each NAND flash memory die 108. For example, there are "N" number of CbAs 309 for "N" number of NAND flash memory die 108. One CbA 309 is coupled to one NAND flash memory die 108, such as Figure 3C As shown. Figure 3C In the embodiment of FIG. 1 , storage device controller 110 includes ECC engine 140. In other embodiments, in addition to using the CbA architecture, Q-learning logic can be coupled to the NAND flash memory die through other architectures, such as other chip stacking architectures.

[0051] for Figures 3A-3C In an embodiment of the Q-learning logic 300, if the selected action is from the set of multiple actions using a default threshold voltage, the Q-learning agent reads data from the NAND flash memory die 108 using the TC parameters of the selected action with the default threshold voltage from the default threshold table 312. If the selected action is from the set of multiple actions using threshold calibration, the RL agent activates the threshold calibration module 310 using the TC parameters of the selected action to calibrate the threshold voltage, and reads data from the NAND flash memory die 108 using the calibrated threshold voltage. The threshold calibration module 310 may calibrate the threshold voltage by valley searching between a distribution of memory states or by a threshold voltage calibration method for memory cells of the NAND flash memory die 108.

[0052] The codeword read using a default threshold voltage or using a calibrated threshold voltage is used by a BER estimation module 314. The BER estimation module 314 estimates the BER of the codeword. The BER estimation module 314 determines the estimated BER based on the number of parity checks that were not met or by other BER estimation methods. If the estimated BER is at or below the decoding limit, the codeword is passed to the ECC engine 140 for decoding. The ECC engine 140 decodes the codeword using any suitable decoding algorithm, such as an iterative low-density parity check (LDPC) algorithm. If the estimated BER is above the decoding limit or if the ECC engine 140 fails to decode the codeword, another read action (e.g., a second read action) that remains to be selected in the current segment of Q learning is selected from the Q table 320 until the segment of the Q learning agent is successfully decoded or times out.

[0053] The Q-table 320 is updated to reflect a positive reward for a selected action that achieves a successful decode, or a negative reward for a selected action that has a higher estimated BER or results in a decode failure. The Q-table 320 can be updated to minimize the cumulative decode latency associated with the estimated BER. Generally, an action with a lower estimated BER will have a low latency. There can be instances where an action with a low latency has a higher Q-value than an action with a lower estimated BER but a high latency.

[0054] Certain embodiments that use Q-learning to select TC parameters for a plurality of environmental states include offline training of TC parameters and online training of TC parameters. In certain embodiments, offline training is performed by a manufacturer of the data storage device 102. In certain embodiments, online training is performed by a particular host 150 in the field using the data storage device 102. In certain aspects, the default threshold voltage facilitates Q-learning to provide TC parameters at the beginning of the life of the data storage device 102. In certain aspects, the calibrated threshold voltage enables Q-learning to provide TC parameters that adapt to changes and variations during the life of the data storage device 102. The Q-learning process of TC with both offline training and online training reduces the cumulative decode latency. The Q-learning process 300 for TC can be implemented on the CbA 309 including the ECC engine 140 (as shown in FIG. 3), on the storage device controller 110 including the ECC engine 140 (as shown in FIG. 4), or on the CbA 309 with the ECC engine 140 included in the storage device controller 110 (as shown in FIG. 5). Figure 3A The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110. Figure 3B The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110. Figure 3C The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110. Figure 3A The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110. 3C The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110. The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110.

[0055] The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110. Figure 4 The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110. The Q-learning logic 300 implemented on the CbA 309 reduces the amount of data transferred to the storage device controller 110 and thus further reduces the cumulative decode latency and reduces the bandwidth usage and power consumption of the storage device controller 110. When the Q-learning logic 300 is implemented on the CbA 309, the determination of the estimated BER can be made without transferring the codeword to the storage device controller 110.

[0056] Each of the plurality of read actions 410 exhibits two TC parameters, where each parameter is individually adjusted to one of three states to set the sense read of the flash memory device. Each TC parameter is an access parameter that reduces temperature-based effects at the data storage device. One example of a TC parameter is bit line voltage (e.g., VBLC). Another example of a TC parameter is sense amplification time (e.g., SEN2) for the duration of a time interval to charge a capacitor for a sense amplifier during a read operation. For example, paramo is a bit line clamp voltage (VBLC) parameter, and parameteri is a SEN2 parameter. VBLC clamps the bit line voltage during sensing, and is typically set to a threshold voltage that is approximately higher than the desired bit line voltage during sensing. Node SEN2 connects the VBLC transistor to the sense bus. As Figure 4 shown, each of the parameters can be adjusted to three states - 1 param, param, and +param. In other embodiments, each of the plurality of read actions can include any number of TC parameters, such as a single parameter or three or more parameters. In other embodiments, each of the plurality of read actions can include parameters adjusted to one of any number of suitable states (e.g., +j on each parameter, respectively).

[0057] As Figure 4As shown, there are eight permutations of param0 and param1 combinations corresponding to read actions A1-A8. Each of read actions A1-A8 has a set of individual default threshold voltages for reading the cell threshold voltages of memory cells of a flash memory device. For example, a set of one default threshold voltage divides two memory states for a single-level cell (SLC), a set of three default threshold voltages divides four memory states for a multi-level cell (MLC), a set of seven default threshold voltages divides eight memory states for a triple-level cell (TLC), or a set of fifteen default threshold voltages divides sixteen memory states for a quad-level cell (QLC). Each set of default threshold voltages for read actions A1-A8 is a predetermined threshold voltage at which the read action performs a read operation without threshold voltage compensation. Threshold voltage compensation generally provides the highest performance for decoding codewords at a low BER. However, through Q-learning, if the recursive Q-learning process determines that the default threshold voltage is at or near the ideal read level, the read action can be performed at the default threshold voltage without conducting the threshold voltage to reduce the overall cumulative decoding latency. The default threshold voltage can be determined by offline characterization of the NAND flash memory die, by using threshold voltages determined by a previous Q-learning segment, by updating the default threshold voltage via Q-learning, or by a combination thereof. For example, the updated default threshold voltage can be a moving average between the old default read threshold and the new threshold voltage from online training. The set of default threshold voltages for read actions A1-A8 can be the same or different across the same state. The set of default threshold voltages for read actions A1-A8 can be the same or different across different states.

[0058] like Figure 4 As shown, there are eight permutations of param0 and param1 combinations corresponding to read actions A9-A16. Each of read actions A9-A16 has a set of calibrated threshold voltages used to read the cell threshold voltages of memory cells of the flash memory device. For example, a set of one calibrated threshold voltage divides two memory states for a single-level cell (SLC), a set of three calibrated threshold voltages divides four memory states for a multi-level cell (MLC), a set of seven calibrated threshold voltages divides eight memory states for a triple-level cell (TLC), or a set of fifteen calibrated threshold voltages divides sixteen memory states for a quad-level cell (QLC).

[0059] The selected read actions with specified TC parameters using default threshold voltages for read actions A1-A8 or calibrated threshold voltages for read actions A9-A16 affect the estimated BER and cumulative decoding delay. Different TC parameters using the same or different set of threshold voltages will affect the estimated BER and cumulative decoding delay.

[0060] When the agent selects one of the plurality of read actions 410 that provides an estimated BER above a decode limit, the agent selects a next read action from the plurality of read actions 410 that is still to be selected in the current Q-learning episode.

[0061] When the agent selects one of the plurality of read actions 410 that provides an estimated BER at or below a decode limit, the agent selects a next action of the decode actions 420 by passing the codeword to the ECC engine 140. If the ECC engine 140 fails to decode the codeword, the agent repeats the process of selecting a next action from the read actions 410 that is still to be selected in the current Q-learning episode.

[0062] If the selected decode action 420 achieves a successful decode, the agent selects a next action of the termination state actions 430 to terminate the Q-learning episode. The termination state actions 430 can also be selected in the event of an expiration of a timeout window without a successful decode.

[0063] The agent uses the Q-table to select and take an action from a plurality of actions for a particular state. The Q-table is updated with a reward resulting from taking the selected action. The reward is a negative decode latency to minimize the latency. If the selected read action has an estimated BER that is too high or fails to achieve a successful decode, another read action is selected from the Q-table without the timeout window expiring. In certain embodiments, if using threshold calibration to select read actions A9-A16 has a high decode failure rate or high latency, the Q-learning process can revert to using default threshold voltages to select read actions A1-A8.

[0064] Figure 5A is a schematic flow diagram of one embodiment of a Q-learning process 500A for TCs to read a flash memory device. For ease of description, the Q-learning process 500A is described with reference to the data storage device 102 of Figure 1 , but any suitable data storage device can be used. The agent of the Q-learning process 500A is executed by the Q-learning logic 300 of Figures 3A-3C .

[0065] At block 510, the agent determines an environmental state of NAND memory cells of the NAND flash memory die 108, such as an environmental state of a page or block of memory cells.

[0066] At block 520, the agent selects an action with the largest Q-value from the Q-table for the current state from the plurality of read actions corresponding to the environmental state determined from block 510 during exploitation, such as a first read action. The agent can select an action without a largest Q-value during exploration (i.e., not during exploitation). Each of the read actions includes one or more TC parameters for a sense read of the flash memory device with a default threshold voltage or a calibrated threshold voltage.

[0067] At block 530, a read operation is performed using the one or more TC parameters of the selected action from block 520 along with a default threshold voltage or a calibrated threshold voltage to perform a read operation to determine a codeword.

[0068] In certain embodiments, a BER estimate of the codeword can be performed to determine an estimated BER. The agent can further determine whether the estimated BER from block 530 is above a decode limit.

[0069] If the BER estimate is at or below the decode limit, the agent can proceed to block 540.

[0070] If the estimated BER is above the decode limit, the agent returns to block 510. The estimated BER from block 530 is part of the environment state of the NAND memory cell used in a subsequent iteration of the Q-learning process 500A at block 510. In the subsequent iteration of the Q-learning process 500A, the agent selects another read action from the plurality of read actions corresponding to the current environment state that is yet to be selected in the current episode of the Q-learning process 500A.

[0071] At block 540, the agent selects a decode action in which an ECC process decodes the codeword from the read operation of block 530. The ECC process can be performed on the CbA 309 of Figure 3A 3C or can be performed on the storage device controller 110 of Figure 3B The agent receives a notification from the ECC engine 140 whether the ECC engine 140 successfully decoded the received codeword.

[0072] If the ECC engine 140 successfully decoded the codeword, the Q-learning process 500A is terminated by selecting and taking a termination state action.

[0073] If the ECC engine 140 did not successfully decode the codeword, the agent returns to block 510. The failed decode from block 540 is part of the environment state of the NAND memory cell used in a subsequent iteration of the Q-learning process 500A at block 510. In the subsequent iteration of the Q-learning process 500A, the agent selects another read action from the plurality of read actions corresponding to the current environment state that is yet to be selected in the current episode of the Q-learning process 500A.

[0074] Figure 5B is a schematic flow diagram of one embodiment of a Q-learning process 500B for TC to read a flash memory device. For ease of description, reference is made to Figure 1 ​The data storage device 102 of FIG. 1 describes a Q-learning process 500B, but any suitable data storage device can be used. The agent of the Q-learning process 500B can be implemented in NAND flash memory logic 109 coupled with each NAND flash memory die 108, or by the storage device controller 110.

[0075] At block 510, the agent determines an environmental state of the NAND memory cells of the NAND flash memory die 108, such as an environmental state of a page or block of memory cells.

[0076] At block 520, the agent selects a read action, such as a first read action, from the plurality of read actions corresponding to the environmental state determined from block 510. Each of the read actions includes one or more TC parameters for a sensing read of the flash memory device with a default threshold voltage or a calibrated threshold voltage.

[0077] At block 530, a sensing operation is performed using the TC parameters of the selected action from block 520 along with the default threshold voltage or the calibrated threshold voltage, the sensing operation to determine a codeword. A BER estimate of the codeword is performed to determine an estimated BER.

[0078] At block 540, the agent determines whether the estimated BER from block 530 is above a decode limit. If the BER estimate is at or below the decode limit, the agent proceeds to block 550.

[0079] If the estimated BER is above the decode limit, the agent returns to block 510. The estimated BER from block 540 is part of the environmental state of the NAND memory cells used in a subsequent iteration of the Q-learning process 500B at block 510. In the subsequent iteration of the Q-learning process 500B, the agent selects another read action from the plurality of read actions corresponding to the current environmental state that is yet to be selected in the current episode of the Q-learning process 500B.

[0080] At block 550, the agent selects and takes a decode action to pass the codeword from the sensing read of block 530 to the ECC engine 140. The ECC engine 140 attempts to decode the received codeword. The ECC process can be performed on the CbA 309 of Figure 3A 、 3C or can be performed on the storage device controller 110 of Figure 3B .

[0081] At block 560, the agent receives a notification from the ECC engine 140 whether the ECC engine 140 successfully decoded the received codeword at block 550.

[0082] If the ECC engine 140 successfully decodes the codeword, the agent proceeds to block 570 in which the segment of the Q-learning process 500B is terminated by selecting and taking a termination state action.

[0083] If the ECC engine 140 does not successfully decode the codeword, the agent returns to block 510. The failed decoding from block 560 is part of the environment state of the NAND memory cell used in the subsequent iteration of the Q-learning process 500A at block 510. In the subsequent iteration of the Q-learning process 500A, the agent selects another read action from the plurality of read actions corresponding to the current environment state that is yet to be selected in the current segment of the Q-learning process 500B.

[0084] In Figure 5A In the schematic illustration of the Q-learning process 500A-B in -B, it is appreciated that each selected action (read action, decode action, termination action, etc.) is an iteration of the recursive Q-learning process. It is appreciated that each selected action changes the environment state of the NAND memory cell. The agent determines the new environment state when selecting a new action.

[0085] After successful decoding or expiration of the timeout window of the Q-learning process 500A-B, the Q-table is updated to reflect the total reward. The total reward (G T ) that minimizes the cumulative decoding latency is the negative sum of the latencies (R) of all selected actions until successful decoding or until expiration of the timeout window. A penalty (F) is added for decoding failure or for expiration of the timeout window. One example of the total reward for decoder success is according to equation II:

[0086] G T = R1+ R2+...+ R T if decoder success (II)

[0087] One example of the reward for decoder failure and timeout expiration is according to equation III.

[0088] G T = R1+ R2+...+ R T if decoder failure and timeout (III)

[0089] In one embodiment, in the case of minimizing average cumulative decode latency, the Q-learning process 500A-B does not use a discount factor because the Q-learning episode is bounded by a timeout window, such that all rewards have the same weight on the reward function. In another embodiment, in the case of minimizing a distribution of current cumulative decode latency relative to future cumulative decode latency, the Q-learning process 500A-B includes a discount factor. In another embodiment, the Q-learning process 500A-B includes a timestamp-dependent customer reward function. For example, exploitation of Q-learning is increased during peak usage times, and exploration of Q-learning is increased during non-peak usage times.

[0090] The plurality of read actions of block 520 includes a plurality of read actions with a default threshold voltage or a calibrated threshold voltage. In online training, the Q-learning process 500A-B learns from updates / approximations of the Q-table by a particular host 150 in the field using the storage device 102. Online training increases reliability and performance by updating the Q-table / approximation of the function of the flash memory device depending on individual user environmental states and / or depending on individual wear and conditions of the flash memory device. Online training adapts to changes in environmental states and individual wear of the flash memory device by continuously updating cumulative rewards. The ability to adapt to changes in environmental states and individual wear of the flash memory device increases performance compared to baseline TC parameters set by the manufacturer. In certain embodiments, online training updates the default threshold voltage so that threshold voltage calibration can be skipped in the next action of the flash memory device in the same or similar environmental state.

[0091] In certain aspects, the Q-learning process achieves reduced cumulative decode latency by selecting TC parameters more quickly compared to parameter search without Q-learning. In certain aspects, the Q-learning process with both online training and offline training achieves reduced cumulative decode latency compared to the Q-learning process with only online training or only offline training.

[0092] Figure 6 is a schematic illustration of an example of certain embodiments of a recursive Q-learning process 600 for TC performed by the storage device controller 110. Figure 7 is a schematic illustration of an example of certain embodiments of a recursive Q-learning process 700 for TC performed by the CbA 109 associated with the NAND flash memory die 108. For ease of description, reference is made to Figure 1 The Q-learning processes 600, 700 are described with respect to implementations in the data storage device 102 of FIG. 1, but the processes can be implemented in any suitable data storage device.

[0093] The two processes 600, 700 show at least one segment of Q-learning for TC. In both processes 600, 700, a Q-table is read from the NVM die 108. In both processes 600, 700, the Q-learning agent selects and takes three read actions to provide codewords at or below a decoding limit. In both processes 600, 700, the Q-learning agent selects and takes a decode action to pass the codeword from the third sense read to the ECC engine 140 of the storage controller 110. In both processes 600, 700, the Q-learning updates the Q-table based on success or failure of the codeword decoding.

[0094] As shown in Figure 6 and Figure 7 , compared to implementing the Q-learning process 600 in the storage controller 110, the amount of data (e.g., codewords) sent to the storage controller 110 is reduced in the Q-learning logic 300 on the CbA 109. Because less data is sent to the storage controller 110 in the Q-learning process 700 of Figure 7 , the Q-learning process 700 implemented on the CbA 109 has lower latency than the Q-learning process 600 implemented on the storage controller 110 of Figure 6 . Further, when multiple NAND flash memory dies 108 are coupled to the same storage controller 110, implementing the Q-learning process 700 on the CbA 109 reduces bandwidth usage of the storage controller 110.

[0095] Embodiments of a data storage device are operable to utilize machine learning (ML) for temperature compensation to read a flash memory device. The machine learning can be implemented in logic coupled to the flash memory die, or can be implemented in the storage controller. The machine learning process determines an environmental state of the flash memory die, and selects one of a plurality of read actions corresponding to the state. Each read action includes at least one TC parameter with a default threshold voltage or with threshold voltage compensation. A read of a codeword is made from the flash memory die using the selected parameter and threshold voltage of the selected read action. If an estimated BER of the codeword is at or below a decoding limit, the codeword is decoded by an error correction code (ECC) process. If the estimated BER of the codeword is above the decoding limit, another read action is selected. The read action is selected based on cumulative rewards of read actions that achieve successful decoding, with a goal of reducing cumulative decoding latency. The ML process selects TC parameters for a plurality of environmental states of a plurality of read operations for the flash memory device with high accuracy and with low cumulative decoding latency by including offline and online training.

[0096] In one embodiment, a method of performing temperature compensation to read a flash memory device includes determining a state of the flash memory device. An action having a maximum Q value from a Q table is selected for the current state during mining. A read operation of a codeword from the flash memory device is performed using one or more parameters according to the selected action. The codeword is decoded with an error correction code (ECC) process.

[0097] In one embodiment, a data storage device includes a memory die. A circuit bounded array (CbA) is coupled to the memory die. The CbA includes Q-learning logic including a BER estimation module and a Q table. The Q-learning logic is operable to determine a current state of the flash memory device; select an action having a maximum Q value from the Q table for the current state during mining; and perform a read operation of a codeword from the memory die using one or more parameters according to the selected action.

[0098] In another embodiment, a data storage device includes a memory die. A controller is coupled to the memory die. The controller includes a BER estimation module; a Q table; and an ECC engine. The controller is operable to determine a current state of the memory die; select an action having a maximum Q value from the Q table for the current state during mining; perform a read operation of a codeword from the memory die using one or more parameters according to the selected action; and decode the codeword with the (ECC) engine.

[0099] Example

[0100] The examples are not intended to limit the scope of the claims except to the extent that the examples are explicitly recited in the claims.

[0101] Example 1

[0102] Figure 8 A sigma plot of the failed bit count (FBC) of reads of a NAND flash memory die in cross-temperature conditions under various searches. Reads of a NAND flash memory die were performed under search 810 on default TC parameters, under search 820 on SEN2 parameters, under search 830 on VBLC parameters, and under search 840 on both SEN2 and VBLC parameters. Search 840 using both TC parameters SEN2 and VBLC exhibited a lower FBC than searches 830, 820, 810.

[0103] Example 2

[0104] Table II shows off-line characterization of NAND flash memory dies at cross-temperatures from 125 °C to -40 °C. The columns are different VBLC parameters. The rows are different SENS 2 parameters. The columns and rows form pairs of VBLC-SENS parameters. The percentage of NAND flash memory blocks of a NAND flash memory die is shown, where the VBLC-SEN2 parameters of a particular pair result in the lowest BER for reads of that particular NAND flash memory block.

[0105]

[0106] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure can be devised without departing from the basic scope thereof, and the scope of the present disclosure is determined by the following claims.

Claims

1. A method of performing temperature compensation to read a flash memory device, comprising: determining a current environmental state of the flash memory device; selecting an action having a maximum Q value from a Q table for the current environmental state during exploitation, the Q table comprising a plurality of actions for each of a plurality of environmental states of the flash memory device; performing a read operation of a codeword from the flash memory device using one or more parameters according to the selected action; and decoding the codeword with an error correction code (ECC) process.

2. The method of claim 1, further comprising selecting an action having a non-maximum Q value from the Q table using a probability of p = 1 - e during exploration.

3. The method of claim 1, further comprising: determining an estimated BER of the codeword from the read operation, wherein the decoding the codeword with the ECC process occurs if the estimated BER is at or below a decoding limit; and selecting another action from the Q table for the current environmental state to perform another read operation of the codeword from the flash memory device using one or more parameters according to the selected other action.

4. The method of claim 1, wherein determining the current environmental state of the flash memory device is a previous read access environmental state in a sequential or random read operation of the flash memory device, wherein the selecting the action determines a next environmental state of the flash memory device.

5. The method of claim 1, wherein a triggering event selected from the group consisting of a temperature difference between a read temperature and a program temperature of stored data being above a threshold, a decoding failure of an ECC engine, a host request, and a recovery mode of a storage device controller is performed prior to the selecting the action.

6. The method of claim 1, wherein the environmental state of the flash memory device comprises one or more characteristics selected from the group consisting of a current environmental state TC parameter, a previous environmental state TC parameter, a current environmental state syndrome weight, a previous environmental state syndrome weight, a time after programming, a program temperature, a read temperature, a number of program / erase cycles, and quantizations thereof.

7. The method of claim 1, wherein the one or more parameters of each of the plurality of actions are selected from the group consisting of a bit line voltage, a sense amplification time, and a read threshold voltage.

8. The method of claim 1, wherein the plurality of actions comprises a first set of actions comprising a first set of one or more parameters determined from an off-line characterization of the flash memory device and a second set of actions comprising a second set of one or more parameters determined from an on-line characterization of the flash memory device.

9. The method of claim 1, wherein the Q table comprises a Q value for each of the plurality of actions, the Q value updated based on a reward to minimize a cumulative decoding latency.

10. A data storage device, comprising: a memory die; a circuit coupled to the memory die, a bounded array (CbA) including Q-learning logic comprising a BER estimation module and a Q-table; and a storage device controller coupled to the CbA; the Q-learning logic operable to: determine a current environmental state of the memory die; select an action having a maximum Q-value from the Q-table for the current environmental state during mining, wherein the Q-table comprises a plurality of actions for each of a plurality of environmental states of the memory die; and perform a read operation of a codeword from the memory die using one or more parameters according to the selected action.

11. The data storage device of claim 10, wherein the CbA further comprises an error correction code (ECC) engine, wherein the Q-learning logic is further operable to pass the codeword to the ECC engine for decoding.

12. The data storage device of claim 10, wherein the storage device controller further comprises an error correction code (ECC) engine, wherein the Q-learning logic is further operable to pass the codeword to the ECC engine for decoding.

13. The data storage device of claim 10, wherein the determined current environmental state of the memory die is a previous read access environmental state of the memory die in a sequential or random read operation, wherein the selected action determines a next environmental state of the memory die.

14. The data storage device of claim 10, wherein the Q-learning logic is for a current read operation following a triggering event selected from the group consisting of: a temperature difference between a read temperature and a program temperature of the stored data being above a threshold, a decoding failure of the ECC engine, a host request, or a recovery mode of the storage device controller.

15. The data storage device of claim 10, wherein the environmental state of the memory die comprises one or more characteristics selected from the group consisting of: a current environmental state TC parameter, a previous environmental state TC parameter, a current environmental state syndrome weight, a previous environmental state syndrome weight, a time after programming, a program temperature, a read temperature, a number of program / erase cycles, and quantizations thereof.

16. The data storage device of claim 10, wherein the one or more parameters of each of the plurality of actions are selected from the group consisting of: a bit line voltage, a sense amplification time, and a read threshold voltage.

17. The data storage device of claim 10, wherein the plurality of actions comprises a first set of actions comprising a first set of one or more parameters determined by an offline characterization of the memory die and a second set of actions comprising a second set of one or more parameters determined by an online characterization of the memory die.

18. The data storage device of claim 10, wherein the Q-table comprises a Q-value for each of the plurality of actions, the Q-value updated based on a reward to minimize a cumulative decoding latency.

19. A data storage device, comprising: a memory die; and a controller coupled to the memory die, the controller comprising: a BER estimation module; a Q table; and an error correction code (ECC) engine; the controller operable to: determine a current environmental state of the memory die; select, during mining, an action having a largest Q value from the Q table for the current environmental state; perform a read operation of a codeword from the memory die using one or more parameters according to the selected action; and decode the codeword with the (ECC) engine.

20. The data storage device of claim 19, wherein the Q table comprises a Q value for each of a plurality of actions and is updated based on a reward to minimize an accumulated decoding latency.

Images