Non-volatile memory die and memory controller with data augmentation components for use with machine learning

Abstract

The present invention discloses methods and apparatus for implementing machine learning data augmentation within a die of a non-volatile memory (NVM) device using on-chip circuit components formed on or within the die. Some specific aspects relate to configuring components under or adjacent to an array of the die to generate augmented versions of images for use in training a deep learning accelerator for an image recognition system by rotating, translating, skewing, cropping, etc. an initial set of training images obtained from a host device. The present invention discloses other methods and apparatus for implementing data augmentation within a storage controller of a data storage device based on machine learning data read from an NVM array of a memory die. Some specific aspects relate to configuring the storage controller to generate augmented versions of training images for use in training a deep learning accelerator for an image recognition system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Patent Application No. 16 / 447,619, filed on June 20, 2019, entitled “NON-VOLATILE MEMORY DIE WITH ON-CHIP DATA AUGMENTATION COMPONENTS FOR USE WITH MACHINE LEARNING,” attorney docket number WDT-1342 (WDA-4383-US), and entitled “STORAGE CONTROLLER HAVING DATA AUGMENTATION COMPONENTS FOR USE WITH NON-VOLATILE MEMORY DIE,” attorney docket number WDT-1342CIP (WDA-4383-1-US), filed on December 17, 2019. 16 / 718,148, a continuation-in-part of U.S. patent application Ser. No. 16 / 447,619, the entire contents of each of which are incorporated herein by reference. Technical Field

[0003] In some aspects, the present disclosure relates to a data storage controller for use with a non-volatile memory (NVM) die. More particularly, but not exclusively, the present disclosure relates to methods and apparatus for implementing data expansion within a data storage controller and / or within a NVM die. Background Art

[0004] Machine learning generally involves the use of artificial intelligence to perform tasks without explicit instructions and instead relying on learned patterns and applying such learning to infer. Deep learning (also referred to as deep structured learning or hierarchical learning) involves machine learning methods based on learning data representations or architectures (such as deep neural networks (DNNs)), rather than task-specific procedures or algorithms. Deep learning is applied to fields such as speech recognition, computer vision, and self-driving vehicles. Deep learning can be implemented or facilitated by a deep learning accelerator (DLA), for example, a microprocessor device designed to accelerate the generation of useful neural networks to implement deep learning.

[0005] A DLA or other machine learning system may need to be trained using initial training data, such as an initial set of images that have been calibrated or labeled for use in training an image recognition system. Data augmentation includes processes for expanding the initial set of images in a realistic but randomized manner to increase the diversity of data used during training. For example, a small set of input images may be slightly altered (by, for example, rotating or skewing the images) to create a larger set of images (i.e., an augmented set of images) for training the system. That is, data augmentation allows calibrated or labeled data to be reused in multiple training instances in order to increase the size of the training data set. Summary of the invention

[0006] A brief overview of some aspects of the disclosure is presented below to provide a basic understanding of these aspects. This overview is not an extensive overview of all contemplated features of the disclosure, and is neither intended to identify key or important elements of all aspects of the disclosure, nor to delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present various concepts of some aspects of the disclosure in a simplified form as a preface to a more detailed description presented later.

[0007] One embodiment of the present disclosure provides a device comprising: a die having a non-volatile memory (NVM) element; and a data expansion controller formed in the die and configured to expand machine learning data stored in the NVM element by expanding the machine learning data.

[0008] Another embodiment of the present disclosure provides a method for use with a die having an NVM array, the method comprising: storing machine learning data within the NVM array of the die; and generating augmented machine learning data using data augmentation circuitry formed in the die.

[0009] Yet another embodiment of the present disclosure provides a device having a die having an NVM array, wherein the device includes: a device formed in the die for storing machine learning data within the NVM array of the die; and a device formed in the die for generating at least one expanded version of the machine learning data.

[0010] Yet another embodiment of the present disclosure provides a data storage device, comprising: a memory die having a non-volatile memory (NVM) element; and a storage controller configured to obtain machine learning data from the NVM element of the memory die and to augment the machine learning data.

[0011] Yet another embodiment of the present disclosure provides a method for use by a storage controller of a data storage device, the method comprising: obtaining machine learning data from an NVM array of a memory die; and generating augmented machine learning data at the storage controller based on the machine learning data.

[0012] Another embodiment of the present disclosure provides an apparatus for use with a data storage device, wherein the apparatus comprises: an apparatus within a storage controller of the data storage device for obtaining machine learning data from an NVM array of a memory die of the data storage device; and an apparatus within the storage controller of the data storage device for augmenting the machine learning data obtained from the NVM array by augmenting the machine learning data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a schematic block diagram illustrating a data storage device in the form of an exemplary solid-state device (SSD) or other data storage device (DSD) having one or more non-volatile memory (NVM) array dies with on-chip data expansion components.

[0014] Figure 2 An example of an NVM die having components under or adjacent to an on-chip array configured for data augmentation processing is shown.

[0015] Figure 3 A flow chart is shown of an exemplary method for performing various types of on-chip data augmentation of image data according to aspects of the present disclosure.

[0016] Figure 4 The NAND array of the NVM die used to store image data is shown, and the Figure 3 Various on-chip data augmentation processes.

[0017] Figure 5 A flow chart of an exemplary method for performing on-chip data augmentation of image data to obtain a noisy image by disabling or at least reducing the use of an on-chip error correction process according to aspects of the present disclosure is shown.

[0018] Figure 6 The NAND array of the NVM die used to store image data is shown, and the Figure 5 The data augmentation process based on error correction.

[0019] Figure 7 A flow chart is shown of an exemplary method for a system in which error correction procedures are performed alternatively by a separate device controller, according to aspects of the present disclosure.

[0020] Figure 8 A flow chart of an exemplary method for performing on-chip data augmentation of an image by adjusting a read voltage during data reading to obtain a noisy image according to aspects of the present disclosure is shown.

[0021] Fig. 9 The NAND array of the NVM die used to store image data is shown, and the Figure 8 The data augmentation process based on the read voltage.

[0022] Fig.10 A flow chart is shown of an exemplary method for performing on-chip data augmentation of an image by storing image data and then reading the image data within a wear region of a NVM die to obtain a noise image in accordance with aspects of the present disclosure.

[0023] Fig.11 The NAND array of the NVM die used to store image data is shown, and the Fig.10 The data augmentation process based on the wear area.

[0024] Fig.12 A flow chart summarizing exemplary on-chip data augmentation operations performed by a NVM die is shown.

[0025] Fig.13 A schematic block diagram configuration for an exemplary NVM device, such as a NAND die, is shown.

[0026] Fig.14 A schematic block diagram providing further details of an exemplary NVM die and its on-chip components is shown.

[0027] Fig.15 A schematic block diagram of an exemplary SSD or other DSD configuration having a controller and one or more NVM arrays is shown, where the controller includes a data expansion component.

[0028] Fig.16 An example of a data storage controller having components configured for data expansion processing is shown.

[0029] Fig.17 A flow chart is shown of an exemplary method for performing various types of data augmentation of image data using a data storage controller according to aspects of the present disclosure.

[0030] Fig.18 A flow chart of an exemplary method for performing data augmentation of image data using a data storage controller by disabling or reducing an error correction routine to obtain a noisy image according to aspects of the present disclosure is shown.

[0031] Fig.19A flow chart of an exemplary method for performing data augmentation of an image using a data storage controller by adjusting a NVM read voltage during data read in order to obtain a noisy image according to aspects of the present disclosure is shown.

[0032] Fig. 20 A flow chart summarizing exemplary data expansion operations performed by a storage controller is shown.

[0033] Fig.21 A schematic block diagram of an exemplary die and a configuration of a storage controller configured to augment machine learning data obtained from the die is shown.

[0034] Fig. 22 A schematic block diagram providing further details of an exemplary apparatus configured to augment machine learning data obtained from a NVM die is shown. DETAILED DESCRIPTION

[0035] In the following detailed description, reference is made to the accompanying drawings which form a part of the detailed description. In addition to the exemplary aspects, embodiments and features described above, other aspects, embodiments and features will become apparent by reference to the accompanying drawings and the following detailed description. The description of the elements in each figure may refer to the elements of the preceding figures. Like numbers may refer to like elements in the accompanying drawings, including alternative embodiments of like elements.

[0036] Examples herein relate to non-volatile memory (NVM) arrays, and to data storage devices or apparatuses for controlling NVM arrays, such as controllers of data storage devices (DSDs) such as solid-state devices (SSDs), and in particular, to solid-state memory storage devices such as those using NAND flash memory (referred to herein as "NAND"). (NAND is a type of non-volatile storage technology that does not require power to retain data. It utilizes negative-AND (i.e., NAND) logic.) For the sake of brevity, an SSD having one or more NAND dies will be used as a non-limiting example of a DSD in the following description of various embodiments. It should be understood that at least some aspects described herein may also be applicable to other forms of data storage devices. For example, at least some aspects described herein may be applicable to data storage or memory devices, including phase change memory (PCM) arrays, magnetoresistive random access memory (MRAM) arrays, and resistive random access memory (ReRAM) arrays. In addition, various embodiments may be used in various machine learning devices, which may include a combination of processing elements and memory / data storage elements, including NVM arrays constructed / configured according to the embodiments.

[0037] Overview

[0038] As described above, machine learning can be implemented or facilitated by a deep learning accelerator (DLA) (e.g., a microprocessor device designed to accelerate the generation of deep neural networks (DNNs) to implement machine learning). These neural networks may also be referred to as learning networks. It may be necessary to train the DLA using initial training data (such as an initial image set for training an image recognition system with a DLA). Data augmentation is the process of modifying the initial image set (in, for example, a realistic but randomized manner) to increase the diversity or variance of the data used during training. For example, a set of input images may be changed (by, for example, rotating or skewing the images) to create a larger image set (e.g., an augmented image set of slightly altered images) for training the system. Data augmentation may be more generally defined as a regularization technique for avoiding overfitting when training a machine learning system (such as a machine learning network or algorithm). Regularization is the process of adding information to solve pathological problems or prevent overfitting during machine learning. For example, regularization may slightly modify the learning model so that the learning model is more effectively generalized based on the training data. In this article, the term data augmentation is defined as generating at least one modified version of the data to avoid or reduce the risk of overfitting during the use of data to train a machine learning system. The data may be, for example, a data vector, a data array, a data object, or a data representation of any number of dimensions, such as a 2-D data object containing one or more patterns. Examples of such data include images or audio clips or other types of digital data, categorical data, time series data, or text.

[0039] Deep learning or machine learning can be implemented using processing components that are integrated with memory components where the data to be processed is stored (i.e., using "near memory" computing) to reduce the need to transfer large amounts of data from one component to another. (Alternative solutions, i.e., using independent processing units such as graphics processing units (GPUs), central processing units (CPUs), etc., and independent memory units such as dynamic random access memory (DRAM), may require transferring large amounts of data from one component to another.)

[0040] Disclosed herein are methods and apparatus for implementing data augmentation for use with near memory machine learning systems (such as DNNs) employing DLAs, wherein data augmentation is performed within the die of the NVM using, for example, under-array data augmentation components or components adjacent to the array, or using components of an off-chip memory controller coupled to the die. That is, disclosed herein is a near memory computing architecture for data augmentation. One or more NVM dies may be part of a DSD such as an SSD.

[0041] It should be noted that a DNN is an example of an artificial neural network with multiple layers between input and output. A DNN is used to determine a mathematical calculation or operation to convert an input into an output, which can be a linear or nonlinear calculation. For example, a DNN can work in its layer by calculating the probability of each output. Each mathematical operation can be considered as a layer. A network with many layers is called having a "deep" layer, hence the term DNN. In a specific example, a DNN may be configured to identify a person within an input image by processing the bits of the input image to produce the identity of a specific person, that is, the output of the DNN is a value that identifies a specific person. It may be necessary to train the DNN. The data augmentation process and apparatus described herein can be used to augment an initial training data set, such as an initial set of labeled images (where a labeled image is an image containing known data, such as an image that has been identified as corresponding to a specific type of object). In addition to configuring the NVM die for near memory data augmentation, the die can also be configured for near memory DNN processing by, for example, providing a DLA on the die and the data augmentation circuit.

[0042] An advantage of at least some of the example methods and apparatus described herein is that only the final results of the data augmentation training process are transmitted to the controller and the host, thereby avoiding the transmission of large amounts of training data, such as an augmented training image set that may include thousands of augmented images.

[0043] It is also noted that the data augmentation machine learning die described herein may be different from a GPU in that a GPU typically transfers computed data from its NVM to a working memory (such as a volatile memory, e.g., RAM / DRAM, or a non-volatile memory suitable for fast access), whereas the augmentation described in various examples herein is performed by the die. (For purposes of simplicity of description, DRAM will be used as the primary and non-limiting example of working memory in the illustrations of various embodiments.) As described above, in some examples, the die includes off-array logic for performing augmentation, storing results, and performing other machine learning operations (such as actual training of the DLA based on augmented data). Thus, in some aspects, a NVM architecture is disclosed that offloads data augmentation from a host device or other device and instead performs augmentation within the NVM die. In addition, at least some of the methods and apparatus disclosed herein exploit the die parallelism and inherent characteristics (such as inherent noise characteristics) of NVM. This may be advantageous for implementing machine learning edge computing applications for on-chip training. Data augmentation may also be performed by a storage controller component using data obtained from the NVM die.

[0044] The data augmentation methods and apparatus described herein may be used in conjunction with the on-chip DLA features and other features described in U.S. patent application Ser. No. 16 / 212,586 and U.S. patent application Ser. No. 16 / 212,596, both of which are entitled “NON VOLATILE MEMORY DIE WITH DEEP LEARNING NEURAL NETWORK” and both of which were filed on Dec. 6, 2018, both of which are assigned to the assignee of the present application and are incorporated herein by reference in their entirety.

[0045] Exemplary machine learning system / process with NVM-based data augmentation

[0046] Figure 1 1 is a block diagram of a system 100 including an exemplary SSD (or DSD) having an NVM with an on-chip learning data augmentation component. The system 100 includes a host 102 and an SSD 104 or other DSD coupled to the host 102. The host 102 provides commands to the SSD 104 for transferring data between the host 102 and the SSD 104. For example, the host 102 can provide a write command to the SSD 104 to write data to the SSD 104, or a read command from the SSD 104 to read data from the SSD 104. The host 102 can be any system or device that requires data storage or retrieval and a compatible interface for communicating with the SSD 104. For example, the host 102 can be a computing device, a personal computer, a portable computer, a workstation, a server, a personal digital assistant, a digital camera, or a digital phone, just to name a few examples. Additionally or alternatively, the host 102 can be a system or device that requires neural network processing for applications such as speech recognition, computer vision, and autonomous vehicles. For example, host computer 102 may be a component of an autonomous driving system for a vehicle.

[0047] The SSD 104 includes a host interface 106, an SSD / DSD controller 108, a volatile memory 110 (such as DRAM) or other working memory, an NVM interface 112 (which may be referred to as a flash memory interface), and an NVM array 114, such as one or more NAND dies configured with an on-chip machine learning data expansion component. The host interface 106 is coupled to the controller 108 and facilitates communication between the host 102 and the controller 108. The controller 108 is coupled to the memory 110 and is coupled to the NVM array 114 via the NVM interface 112. The host interface 106 can be any suitable communication interface, such as a non-volatile memory express (NVMe) interface, a universal serial bus (USB) interface, a serial peripheral (SP) interface, an advanced technology attachment (ATA) or serial advanced technology attachment (SATA) interface, a small computer system interface (SCSI), an IEEE 1394 (FireWire) interface, etc. In some embodiments, the host 102 includes the SSD 104. In other embodiments, SSD 104 is remote from host 102, or is contained in a remote computing system that is communicatively coupled to host 102. For example, host 102 may communicate with SSD 104 via a wireless communication link.

[0048] Although in Figure 1 In the example shown, SSD 104 includes a single channel between controller 108 and NVM die 114 via interface 112, the subject matter described herein is not limited to having a single memory channel. For example, in some NAND memory system architectures, two, four, eight or more NAND channels couple the controller and the NAND memory devices, depending on the capabilities of the controller. In any of the embodiments described herein, even if a single channel is shown in the figures, more than a single channel may be used between the controller and the memory die. Controller 108 may be implemented in a single integrated circuit chip and may communicate with different layers of memory in NVM die 114 via one or more command channels.

[0049] Controller 108 controls the operation of SSD 104. In various aspects, controller 108 receives commands from host 102 via host interface 106 and executes the commands to transfer data between host 102 and NVM array 114. In addition, controller 108 can manage reading and writing to memory 110 to perform various functions implemented by the controller, and maintain and manage cached information stored in memory 110.

[0050] Controller 108 may include any type of processing device, such as a microprocessor, a microcontroller, an embedded controller, logic circuits, software, firmware, etc., for controlling the operation of SSD 104. In some aspects, some or all of the functions described herein as being performed by controller 108 may alternatively be performed by another element of SSD 104. For example, SSD 104 may include a microprocessor, a microcontroller, an embedded controller, logic circuits, software, firmware, or any type of processing device for performing one or more of the functions described herein as being performed by controller 108. According to other aspects, one or more of the functions described herein as being performed by controller 108 may alternatively be performed by host 102. In further aspects, some or all of the functions described herein as being performed by controller 108 may alternatively be performed by another element, such as a controller in a hybrid drive that includes both non-volatile memory elements and magnetic storage elements.

[0051] The memory 110 may be any suitable memory, computing device, or system capable of storing data. For example, the memory 110 may be a normal RAM, a DRAM, a double data rate (DDR) RAM (DRAM), a static RAM (SRAM), a synchronous dynamic RAM (SDRAM), a flash memory device, an erasable programmable read-only memory (EPROM), an electrically erasable programmable ROM (EEPROM), etc. In various embodiments, the controller 108 uses the memory 110 or a portion thereof to store data during data transfer between the host 102 and the NVM array 114. For example, the memory 110 or a portion of the memory 110 may be a cache memory. The NVM array 114 receives data from the controller 108 via the NVM interface 112 and stores the data. The NVM array 114 may be any suitable type of non-volatile memory, such as a NAND-type flash memory, etc. In some embodiments, the volatile memory 110 may be replaced by a non-volatile memory such as MRAM, PCM, ReRAM, etc. to be used as a working memory for the entire device.

[0052] exist Figure 1 In an example of the present invention, controller 108 may include hardware, firmware, software, or any combination thereof, which provides a machine learning controller 116 for use with NVM array 114 (wherein in some examples, the machine learning controller may include at least some off-chip data expansion components, such as components that control data expansion based on controlling off-chip error correction). Although Figure 1An exemplary SSD is shown and SSDs are generally used as illustrative examples throughout the specification, but the various embodiments disclosed are not necessarily limited to SSD applications / implementations. As an example, the disclosed NVM die and associated processing components may be implemented as part of a package that includes other processing circuits and / or components. For example, a processor may include the embedded NVM described herein and associated circuits and / or components for machine learning, or otherwise be coupled to the embedded NVM described herein and associated circuits and / or components. As an example, the processor may offload certain machine learning tasks to the NVM and associated circuits and / or components. As another example, the controller 108 may be a controller in another type of device and still include the machine learning controller 116 and perform some or all of the functions described herein.

[0053] Figure 2 A block diagram of an exemplary NVM die 200 is shown that includes an NVM storage array component 202 and a processing component 204 below or adjacent to the array (or otherwise outside the array). Not all circuits or memory components that may be used in an actual NVM die, such as input and output components, voltage regulation components, clock and timing components, etc., are shown. Instead, only some components and circuits are shown, summarized as a block diagram or schematic. The exemplary NVM array component 202 includes: an NVM storage device 206 for storing machine learning training data such as input image data and augmented image data; and an NVM storage device 208 configured to store other data (such as DNN synaptic weights, bias values, etc.) or other types of user data or system data.

[0054] The NVM off-array processing unit 204 includes a data expansion unit 210 configured to perform or control a data expansion operation. Figure 2In the example of , the exemplary data augmentation component 210 includes: one or more noise adding components 212, which are configured to generate augmented machine learning data by adding noise to the initial machine learning data, such as by adding irrelevant noise to each of the initial labeled training image set; one or more skewing components 214, which are configured to generate augmented machine learning data by skewing the initial machine learning data, such as by skewing each of the initial labeled training image set in a different way; one or more cropping components 216, which are configured to generate augmented machine learning data by cropping the initial machine learning data, such as by cropping each of the initial set of labeled training images in a different way; one or more flip / rotate / translate components 218, which are configured to generate augmented machine learning data by flipping, rotating and / or translating the initial machine learning data, such as by flipping, rotating and / or translating each of the initial labeled training image set in a different way.

[0055] Multiple instances of each augmentation component (212, 214, 216, and 218) are shown, so in some examples, multiple such devices may operate in parallel. For example, N noise addition components 212 may be provided to process N different input training images simultaneously, thereby generating an augmented image set based on each of the N different input training images. In other examples, only a single instance of each component may be provided. In other examples, only one or a few of the components shown are provided, such as only a noise addition component 212 or only a skew component 214. In other examples, other augmentation components are provided in addition or alternatively, which are used to augment the initial data set in other ways. It should also be noted that Figure 2 The exemplary components of mainly relate to the expansion of image data. For examples where the data is not image data but is, for example, audio data, different expansion components suitable for the data type may be provided.

[0056] Figure 2The NVM off-array processing component 204 also includes various other components, including: a machine learning value storage controller 226, which is configured to store machine learning data in the NVM storage device 206; a machine learning value read controller 228, which is configured to read previously stored machine learning data from the NVM storage device 206; and an on-chip error correction code (ECC) controller 230, which is configured to control any on-chip ECC applied to data to address bit error rate (BER) when it reads data from the NVM array component 202. As will be explained, certain types of data augmentation can be performed by adjusting the ECC or, in some cases, disabling the ECC, thereby increasing the BER to selectively add noise to the image. That is, the data augmentation controller can be configured to generate augmented data by reducing the amount of error correction performed by the error correction component compared to the average amount of error correction that would otherwise be used to read data not subject to data augmentation, and then reading the stored data from the NVM element with the reduced error correction. Thus, in one example, if a device normally checks three bytes of ECC data within 512 bytes of data, the ECC can be reduced by checking only two bytes of ECC data. Thus, if the die itself is equipped to perform ECC processes, those processes can be disabled or modified to increase the noise in the data read from the NAND array, thereby providing augmented image data. Note that in many systems, ECC is instead performed by a device controller separate from the die (such as a Figure 1 208). In such a specific implementation, the die itself does not control ECC and therefore cannot adjust ECC directly. In such a system, because ECC is performed by the controller, all images read by the die can be "noise" images suitable for use as augmented images in on-chip training. In addition, it should be noted that the BER of the NVM block can vary depending on the underlying conditions and memory type, and therefore the read controller 228 can be programmed or configured to take such information into account when selecting a target location for storing data. For example, for data augmentation purposes, write parameters can be selected or modified to increase the BER so that any augmentation requirements are met. This contrasts with the conventional desire to reduce the BER to make ECC decoding easier. As will be explained below, one technique for modifying the write parameters is to modify the location where the image data is written so that the data is stored in a worn area of ​​the NVM array 202, thereby increasing storage errors so as to inject noise into the stored / retrieved image data.

[0057] Figure 2Also shown is a machine learning controller 232, which can be, for example, a DLA, a DNN, a pattern recognition controller, an image recognition controller, etc., configured to perform some form of machine learning using the augmented data. In some examples, the augmented data is stored in the NVM array 202 for later use. In other examples, the augmented data is maintained in other memory within the die, such as a data cache ( Figure 2 ), for immediate use by the training component, and then erased or rewritten. That is, in some examples, the augmented data can be transient data that is saved only when it is needed to train the machine learning system (e.g., DNN) and then discarded.

[0058] In the following, various exemplary data augmentation systems and processes are described, where data is stored in a NAND array, and where data augmentation is used to train an image or pattern recognition system. As already explained, other types of NVM arrays may be used, and data augmentation may be applied to other types of machine learning. Thus, the following description provides illustrative and non-limiting examples.

[0059] Image recognition example using NVM-based data augmentation

[0060] Figure 3 An exemplary method 300 for data augmentation for use with image recognition according to aspects of the present disclosure is shown, wherein any of the aforementioned forms of data augmentation (e.g., skewing, rotating, etc.) may be applied. Beginning at block 302, input circuitry of an NVM die inputs an initial set of labeled (or calibrated) input images for use with image recognition training (or for use with other forms of deep learning or machine learning), and stores the initial set of images within a NAND NVM array of the die. At 304, read circuitry of the NVM die reads one or more of the labeled images from the NAND NVM array. At 306, the data augmentation circuitry of the NVM die generates a set of altered versions of the labeled images by, for example, rotating, translating, skewing, cropping, flipping, and / or adding noise to the labeled images read from the NAND NVM, thereby providing an augmented set of images.

[0061] At 308, machine learning circuitry of the NVM die uses the augmented image set to perform machine learning (such as DLA learning) to, for example, train an image recognition system to recognize images within the augmented image set, and then output a set of trained parameters. In some examples, the parameters may include synaptic weights, bias values, etc. for use with a DNN configured for image recognition. The image recognition system itself may be configured, for example, within circuitry off-array of the die, or may be embodied elsewhere, such as in an SSD controller, a host system, or a remote server.

[0062] As far as flipping is concerned, when using a DLA, it is usually desirable to store the image in an analytical format (rather than a compressed format like JPEG). With an analytical image, flipping of the image can be achieved by reversing the order of the pixels read. If the dimensions and parameters of the image are stored in NAND memory (as is the case with an on-chip DLA), and the parameters are therefore available to the die logic for flipping, flipping on different axes can be performed by the die. Note also that noise can be added to the image by omitting every other bit of the image, or every other row or column of the image, or by performing other relatively straightforward adjustments to the image to generate a "noisy" version of the image.

[0063] Figure 4 An NVM die (such as Figure 2 NAND array 400 of a die) and using Figure 3 At 402, a host device or other external system provides a labeled training image, such as a labeled image of a particular individual to be recognized by an image recognition system or a particular type of object or animal to be detected. At 404, circuitry of the NVM die inputs and stores the labeled image within a first portion or region 406 of the NAND array 400. Later, when a data augmentation process is initiated, circuitry of the NAND die reads the labeled image at 408, modifies the image at 410 to generate an augmented set of labeled images, and then stores the augmented set of images in a second region or portion 414 of the NAND array 400 at 412 for subsequent use in training an image recognition system, such as a DLA for training an on-chip image recognition system. Additionally or alternatively, at 416, circuitry of the NAND die (substantially) immediately uses the augmented set of images to train the image recognition system, such as by applying the augmented set of images directly to the on-chip DLA.

[0064] Figure 5An exemplary method 500 for data augmentation for use with image recognition according to aspects of the present disclosure is shown, wherein data augmentation is performed by disabling or at least reducing the use of on-chip ECC (or other on-chip error correction systems or processes). Note that in this article, disabling ECC is an example of reducing the use of ECC. Starting from box 502, the input circuit of the NVM die inputs an initial set of labeled input images for use with image recognition training (or other forms of deep learning or machine learning), and the initial set of labeled input images is stored in the NAND NVM array of the die, where the die is configured to apply on-chip error correction to data read from the NAND NVM array. At 504, the control circuit of the NVM die disables the on-chip error detection and correction process, or otherwise reduces the amount (or effectiveness) of on-chip error correction applied by the NVM to the data read from the array, thereby selectively increasing the effective BER. By disabling ECC, the data is read "as is", i.e., without DCC-based decoding. This reduces latency and saves power while also producing a noisy image for data augmentation. At 506, read circuitry of the NVM die repeatedly reads images from the NAND NVM array without or with reduced on-chip error correction to generate an augmented set of labeled images that differ from each other and from the initial image due to different noise artifacts caused by the lack of on-chip error correction or the reduced error correction. In this way, an augmented data set with uncorrelated noise artifacts (e.g., different noise vectors) can be generated using inherent noise associated with the natural BER of the die. At 508, machine learning circuitry of the NVM die performs machine learning (such as DLA learning) using the augmented set of labeled images to, for example, train an image recognition system to recognize images within the set, and then output a trained set of parameters. The image recognition system may be configured, for example, within an off-array circuit of the die, or may be embodied elsewhere, such as in an SSD controller, a host system, or a remote server.

[0065] Figure 6 An NVM die (such as Figure 2 NAND array 600 of a die) and using Figure 5At 602, a host device or other external system provides a labeled training image. At 604, the circuitry of the NVM die inputs and stores the labeled image in a first portion or region 606 of the NAND array 600. Later, when a data augmentation process is initiated, the circuitry of the NAND die: repeatedly reads the labeled image with ECC disabled or reduced at 608 to provide or maintain noise within the read image, thereby generating an augmented set of labeled images; and then stores the augmented set of images in a second region or portion 612 of the NAND array 600 at 610 for subsequent use in training an image recognition system. Additionally or alternatively, at 614, the circuitry of the NAND die (substantially) immediately uses the augmented set of images to train the image recognition system.

[0066] exist Figure 6 , multiple arrows emanating from the first array portion 606 are shown to emphasize that individual images stored therein may be repeatedly read. Each individual read from the NAND array performed without on-chip ECC or with reduced on-chip ECC will typically result in different noise artifacts in the readout image, where the noise artifacts are uncorrelated with each other, thereby providing noise-based data augmentation or a noise-augmented data set. The read operation may be repeatedly performed until a training system using the augmented data set is satisfied that a sufficient number of samples of each particular image have been collected, such as by comparing the number of samples to a suitable threshold or by verifying that the system has been adequately trained. In some examples, a read channel or NVM device controller separate from the NVM die (i.e., off-chip) may be configured to perform Figure 5 and Figure 6 at least some of the processes or operations of the die, for example, if the ECC is performed by a device or component separate from the die.

[0067] As mentioned above, in some systems, ECC is provided by a device controller separate from the NAND die (such as Figure 1 Controller 108) executes. Figure 7 Summarize the method that can be performed by the die. In short, at block 702, the die reads stored (target) data (which can be image data for DLA training or can be other data) from the NAND block. At decision block 704, the die determines whether data augmentation mode is on. If data augmentation mode is on, then at block 706, the die uses the read data in DLA training or other machine learning training. If data augmentation mode is off, then at block 708, the die instead sends a request to a controller (such as a processor). Figure 1 A separate device controller 108 or a controller formed on the NAND) sends the read data to perform ECC decoding on the data so that the data can then be processed normally.

[0068] Figure 8 An exemplary method 800 for data augmentation for use with image recognition according to aspects of the present disclosure is shown, wherein data augmentation is performed by modifying a read voltage to inject noise into a read image (or otherwise obtain a greater number of read errors). Beginning at block 802, an input circuit of an NVM die inputs an initial set of labeled input images for use with image recognition training (or other forms of deep learning or machine learning), and the initial set of labeled input images is stored within a NAND NVM array of the die. At block 804, a control circuit of the NVM die identifies a read voltage for reading data from the NVM array with minimal read errors (e.g., set to a normal read voltage that achieves a low BER). At block 806, the control circuit of the NVM die modifies the read voltage applied to its NVM elements (compared to a read voltage that would otherwise be used to read an image not subject to data augmentation (e.g., the voltage with minimal read errors identified at block 804)). Thus, in one example, if an average threshold voltage of X volts is typically used to read data, the modified read voltage may be 0.9X volts. At block 808, the read circuitry of the NVM die applies the modified read voltage to the NVM element while reading one or more initial labeled images from the NVM element to generate a set of augmented labeled images that differ from each other and from the initial images due to different noise artifacts caused by the modified read voltage. At 810, the machine learning circuitry of the NVM die performs machine learning (such as DLA learning) using the augmented set of labeled images to, for example, train an image recognition system to recognize images within the set, and then outputs a set of trained parameters.

[0069] Fig. 9 An NVM die (such as Figure 2 NAND array 900 of a die) and using Figure 8 Methods for manipulating and processing data in various processes. At 902, a host device or other external system provides a labeled training image. At 904, the circuitry of the NVM die inputs and stores the labeled image within a first portion or region 906 of the NAND array 900. Later, when a data augmentation process is initiated, the circuitry of the NAND die: repeatedly reads the labeled image at 908 with a modified read voltage so as to provide or maintain noise within the read image, thereby generating an augmented set of labeled images; and then stores the augmented set of images into a second region or portion 912 of the NAND array 900 for subsequent use in training an image recognition system. Additionally or alternatively, at 914, the circuitry of the NAND die (substantially) immediately uses the augmented set of images to train the image recognition system. Fig. 9In the figure, multiple arrows are shown leading from the first array portion 906 to emphasize that individual images stored therein can be repeatedly read by possibly different read voltages. Each individual read from the NAND array will typically result in different noise artifacts, where the noise artifacts are uncorrelated with each other, thereby providing noise-based data augmentation or a noise-augmented data set.

[0070] Fig.10 An exemplary method 1000 for data augmentation for use with image recognition according to aspects of the present disclosure is shown, wherein data augmentation is performed by repeatedly writing (initially un-augmented) data to a worn region of an NVM and then reading data from the worn region of the NVM, thereby injecting noise into the image. Beginning at block 1002, an input circuit of an NVM die inputs an initial set of labeled input images for use with image recognition training (or other forms of deep learning or machine learning), and stores the initial set of labeled input images within a NAND NVM array of the die, wherein the die has regions affected by different amounts of wear. At 1004, a control circuit of the NVM die identifies a worn region of the NVM array that is subject to storage errors. Any suitable technique may be used to identify the worn region of the NVM array, such as by tracking the BER of data read from various blocks. At block 1006, a read circuit of the NVM die reads the labeled image from the initial storage region of the NVM array, and at block 1008, a write (programming) circuit of the NVM die re-stores the labeled image in the worn region of the NVM that is subject to storage errors. At block 1010, the read circuitry of the NVM die re-reads the marker image from the worn region of the NVM array, thereby obtaining a noise augmented version of the marker image, wherein the noise is caused by storing / reading from the worn region of the NVM array with a high BER. At 1012, the machine learning circuitry of the NVM die performs machine learning (such as DLA learning) using the augmented set of marker images to, for example, train an image recognition system to recognize the image, and then outputs a set of trained parameters.

[0071] Fig.11 An NVM die (such as Figure 2 NAND array 1100 of a die) and using Fig.10Various processes for manipulating and processing data using methods. At 1102, a host device or other external system provides a labeled training image. At 1104, the circuitry of the NVM die inputs and stores the labeled image within a first (unworn) region 1106 of the NAND array 1100. Later at 1108, when a data augmentation process is initiated, the circuitry of the NVM die re-stores the labeled image in a worn region 1110 of the NAND array. At 1112, the read circuitry of the NVM die repeatedly reads the labeled image from the worn region 1110, thereby generating an expanded set of labeled images using uncorrected read errors. At 1114, the circuitry of the NVM die stores the expanded set of images in another region 1116 of the NAND array 1100 for subsequent use in training an image recognition system. Additionally or alternatively, at 1118, the circuitry of the NAND die (substantially) immediately uses the expanded set of images to train the image recognition system. Fig.11 In the figure, multiple arrows leading from the worn array portion 1110 are shown to emphasize that individual images stored therein can be read repeatedly. Due to the wear characteristics of the array area 1110, each individual read from the NAND array will generally result in different noise artifacts, where the noise artifacts are uncorrelated with each other, thereby providing a noise-enlarged data set.

[0072] In the following, various general exemplary processes and systems for on-chip (NVM-based) data augmentation are described.

[0073] Additional Exemplary Methods and Apparatus Employing NVM-Based Data Augmentation

[0074] Fig.12 A process 1200 is generally shown in accordance with some aspects of the present disclosure. Process 1200 may occur within any suitable apparatus or device having a die capable of performing operations, such as a NAND die. At box 1202, a memory die (e.g., a suitably configured NAND die) stores machine learning data within an NVM array of the memory die. At box 1204, the die generates augmented machine learning data using data augmentation circuitry formed in the memory die or using components of a memory controller. Examples are described above. With respect to using components of a memory controller, by way of example, an ECC component of a memory controller may be configured or controlled to allow or facilitate the creation of an augmented data set by disabling or reducing the ECC.

[0075] Fig.13An embodiment of an apparatus 1300 configured in accordance with one or more aspects of the present disclosure is generally shown. Apparatus 1300 or components thereof may be embodied or implemented within a NAND die or some other type of NVM device supporting data storage. Apparatus 1300 includes an NVM element 1302 and a data expansion controller 1304 configured to expand machine learning data stored within NVM element 1302 using expanded machine learning data. Examples of apparatus are described above. Additional examples are described below. As described above, at least some data expansion components may be separate from the die, such as an ECC component of a NAND device controller.

[0076] Fig.14 An embodiment of an apparatus 1400 configured in accordance with one or more aspects of the present disclosure is shown. The apparatus 1400 or components thereof may be embodied or implemented within a NAND die or some other type of NVM device that supports data storage. In various implementations, the apparatus 1400 or components thereof may be a processor, a controller, a computing device, a personal computer, a portable device, or a workstation, a server, a personal digital assistant, a digital camera, a digital phone, an entertainment device, a medical device, an autonomous vehicle control device, or a component of any other electronic device that stores, processes, or uses neural data.

[0077] The device 1400 includes a communication interface 1402, a physical memory array (e.g., a NAND block) 1404, and off-array processing circuits 1410, 1411 (e.g., circuits below or adjacent to the array). These components may be Fig.14 Suitable components generally indicated by the connecting lines in FIG. are coupled to each other and / or placed in electrical communication with each other. Although not shown, other circuits may be provided, such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further.

[0078] The communication interface 1402 provides a component for communicating with other devices via a transmission medium. In some specific implementations, the communication interface 1402 includes circuits and / or programming (e.g., programs) suitable for facilitating bidirectional transmission of information relative to one or more devices in the system. In some embodiments, the communication interface 1402 may be configured for wired-based communication. For example, the communication interface 1402 may be a bus interface, a send / receive interface, or some other type of signal interface, including a circuit for outputting and / or obtaining a signal (e.g., outputting a signal from an SSD and / or receiving a signal into an SSD). The communication interface 1402 is used as an example of a component for receiving and / or a component for transmitting.

[0079] The physical memory array 1404 may represent one or more NAND blocks. The physical memory array 1404 may be used to store data, such as images, that are manipulated by the circuits 1410, 1411 or some other component of the device 1400. The physical memory array 1404 may be coupled to the circuits 1410, 1411 so that the circuits 1410, 1411 may read or sense information from the physical memory array 1404 and write or program information to the physical memory array. That is, the physical memory array 1404 may be coupled to the circuits 1410, 1411 so that the physical memory array 1404 may be accessed by the circuits 1410, 1411.

[0080] Circuits 1410, 1411 are arranged or configured to obtain, process and / or send data, control data access and storage, issue or respond to commands, and control other desired operations. For example, circuits 1410, 1411 may be implemented as one or more processors, one or more controllers, and / or other structures configured to perform functions. According to one or more aspects of the present disclosure, circuits 1410, 1411 may be adapted to perform any or all of the off-array features, processes, functions, operations, and / or routines described herein. For example, circuits 1410 may be configured to perform operations relative to Figures 2 to 13 As used herein, the term "adapted to" with respect to processing circuits 1410, 1411 may refer to the circuit being configured, employed, implemented and / or programmed to perform one or more of the specific processes, functions, operations and / or routines according to the various features described herein. The circuit may include a dedicated processor (such as an application specific integrated circuit (ASIC)) that is used to perform the combined Figures 2 to 13 A means for performing any of the operations described herein (e.g., a structure for performing the operation). Circuitry is used as an example of a means for processing. In various specific implementations, circuitry may at least partially provide and / or incorporate components described above for various embodiments (including, for example, Figure 2 The function described by element 204).

[0081] According to at least one example of the device 1400, the processing circuits 1410, 1411 may include one or more of the following: a circuit / module 1420 configured to store an image or other machine learning data in a NAND block; a circuit / module 1422 configured to read an image or other machine learning data from the NAND block; a circuit / module 1424 configured to control expansion of the image or other machine learning data; a circuit / module 1426 configured to skew an image; a circuit / module 1428 configured to crop an image; a circuit / module 1430 configured to flip / rotate / translate an image; a circuit / module 1432 configured to expand the image via noise control; charging; a circuit / module 1433 configured to perform ECC; a circuit / module 1434 configured to disable ECC; a circuit / module 1436 configured to reduce ECC; a circuit / module 1437 configured to control a read voltage; a circuit / module 1438 configured to adjust the read voltage to inject noise; a circuit / module 1439 configured to control machine learning through initial data and augmented data; a circuit / module 1441 configured to identify a worn NVM area; and a circuit / module 1443 configured to store data to and / or read data from a worn NVM area storage / read component.

[0082] like Fig.14 As shown, the physical memory array 1404 may include one or more of the following: a block 1440 for storing machine learning data (such as an input labeled image); a block 1442 for storing an extended version of the machine learning data; a block 1444 that serves as a wear area; and a block 1446 for storing other user data or system data (e.g., data related to the overall control of the operation of the NAND die).

[0083] In at least some examples, a method for performing Fig.141420; means for reading the image or other machine learning data from the NAND block, such as circuit / module 1422; means for controlling the augmentation of the image or other machine learning data, such as circuit / module 1424; means for skewing the image, such as circuit / module 1426; means for cropping the image, such as circuit / module 1428; means for flipping / rotating / translating the image, such as circuit / module 1430; means for augmentation via noise control, such as circuit / module 1432; means for performing ECC, such as circuit / module 1433; means for disabling ECC, such as circuit / module 1434; means for reducing ECC, such as circuit / module 1436 ; means for controlling the read voltage, such as circuit / module 1437; means for adjusting the read voltage to inject noise, such as circuit / module 1438; means for controlling machine learning through initial data and augmented data, such as circuit / module 1439; means for identifying worn NVM areas, such as circuit / module 1441; means for storing data to and / or reading data from worn NVM area storage / read components, such as circuit / module 1443; means for storing machine learning data, such as NAND block 1440; means for storing an augmented version of the machine learning data, such as NAND block 1442; and means for storing other user data and / or system data (e.g., data relating to the overall control of the operation of the NAND die), such as NAND block 1446.

[0084] In other examples, means are provided for storing machine learning data within an NVM array of a die, such as formed in a die. Figure 2 NVM element 1202; and providing means for generating at least one expanded version of the machine learning data, such as also formed in the die Figure 2 The data expansion component 210 is provided.

[0085] Referenced Figures 1 to 14 Described are illustrative embodiments in which data augmentation is performed by an on-chip NVM die component.In the following, illustrative embodiments are described in which data augmentation is performed by a storage controller component using data obtained from a NVM die.

[0086] Exemplary method / apparatus employing storage controller-based data augmentation

[0087] Fig.151 is a block diagram of a system 1500 including an exemplary SSD or other DSD having a storage controller with a machine learning data augmentation component. The system 1500 includes a host 1502 and an SSD or other DSD 1504 coupled to the host 1502. Many components of the system 1500 are similar to Figure 1 1504 is a component of system 100 and will not be described in detail. SSD 1504 includes a host interface 1506, a controller 1508, volatile or working memory 1510 (e.g., DRAM), an NVM interface 1512 (e.g., a flash memory interface), and an NVM array 1514 (having one or more dies) for storing machine learning data such as training images for a DNN.

[0088] exist Fig.15 In an example of the present invention, controller 1508 includes a machine learning data augmentation system 1518 in the form of hardware, firmware, software, or any combination thereof that performs data augmentation of machine learning data obtained from NVM array 1514 (wherein in some examples, NVM array 1514 may include at least some on-chip data augmentation components on the die, such as those described above in conjunction with Figures 1 to 14 discussed). Fig.15 An exemplary SSD is shown and is generally used as an illustrative example in these descriptions, but the various embodiments disclosed are not necessarily limited to SSD applications / implementations. Other DSD examples are mentioned above. In one or more embodiments, the controller 1508 is also responsible for performing data storage and data management functions, such as performing various read and write operations in response to commands from the host 1502.

[0089] In examples with two or more memory dies, machine learning data stored on the various NVM dies can be transmitted in parallel to the controller 1508 for use by its data expansion components. In this way, multiple images (or other types of machine learning input data) can be provided to the controller 1508 simultaneously (e.g., substantially simultaneously) and in parallel by multiple NVM dies. Parallelism can improve overall processing efficiency, such as providing faster processing.

[0090] Fig.16 A block diagram of an exemplary machine learning data augmentation system 1518 for a data storage controller 1508 is shown. Fig.16 Also shown is working memory 1510 (e.g., RAM or DRAM), which provides temporary storage for (1) machine learning training data (such as input image data received from NVM array 1514) and (2) augmented image data generated by data augmentation system 1518. Fig.16 Also shown is an exemplary NVM array 1514 that stores machine learning training data such as input image data.

[0091] The data expansion system 1518 includes a data expansion component 1610 configured to perform or control data expansion operations. Fig.16 In the example of , the exemplary data augmentation component 1610 includes one or more of the following: one or more noise adding components 1612, the one or more noise adding components are configured to generate augmented machine learning data by adding noise to the initial machine learning data, such as by adding uncorrelated noise to each of the initial labeled training image set; one or more skew components 1614, the one or more skew components are configured to generate augmented machine learning data by skewing the initial machine learning data, such as by skewing each of the initial labeled training image set in a different manner; one or more cropping components 1616, the one or more cropping components are configured to generate augmented machine learning data by cropping the initial machine learning data, such as by cropping each of the initial set of labeled training images in a different manner; one or more flip / rotate / translate components 1618, the one or more flip / rotate / translate components are configured to generate augmented machine learning data by flipping, rotating and / or translating the initial machine learning data, such as by flipping, rotating and / or translating each of the initial labeled training image set in a different manner. These components can be the same or similar to the corresponding components of the on-chip data augmentation component described above.

[0092] Multiple instances of each augmentation component (1612, 1614, 1616, and 1618) are shown, so in some examples, multiple such devices may operate in parallel. For example, N noise adding components 1612 may be provided to process N different input training images simultaneously, thereby generating an augmented image set based on each of the N different input training images. In other examples, only a single instance of each component is provided. In still other examples, only one or a few of the components shown are provided. In still other examples, other augmentation components are provided additionally or alternatively. It should also be noted that Fig.16 The exemplary components of mainly relate to the expansion of image data. For examples where the data is not image data but is, for example, audio data, different expansion components suitable for the data type may be provided.

[0093] Fig.16The data augmentation system 1518 of the data storage controller includes various other components, such as: a machine learning value storage controller 1626, which is configured to store machine learning data in the working memory 1510 and / or NVM 1514; a machine learning value read controller 1628, which is configured to read previously stored machine learning data from the NVM array 1514; and an ECC controller 1630, which is configured to control any ECC applied to data obtained from the NVM array 1514 to address BER or other issues. As already explained, certain types of data augmentation can be performed by adjusting the ECC or, in some cases, disabling the ECC, thereby increasing the BER to selectively add noise to the image. That is, the data augmentation system 1518 of the data storage controller can be configured to generate augmented data by reducing the amount of error correction performed by the error correction component compared to the average amount of error correction that would otherwise be used to read data not subject to data augmentation, and then reading the stored data from the NVM array 1514 with the reduced error correction. If the die itself is equipped to perform ECC processes, those processes may be disabled or modified by the data augmentation system 1518 to increase the noise in the data read from the NAND array, thereby providing augmented image data.

[0094] Fig.16 Also shown is a machine learning controller 1632, which can be, for example, a DLA, a DNN, a pattern recognition controller, an image recognition controller, etc., configured to use the augmented data to perform some form of machine learning. In some examples, the augmented data is stored in the NVM array 1514 for later use. In other examples, the augmented data is maintained in the working memory 1510 for use by the training component and then erased or rewritten. That is, in some examples, the augmented data can be transient data that is saved only when it is needed to train the machine learning system (e.g., DNN) and then discarded.

[0095]

[0013] In the following, various exemplary data augmentation systems and processes are described in which data augmentation performed by a data storage controller is used to train an image or pattern recognition system.

[0096] Fig.17An exemplary method 1700 of a data augmentation operation performed by a data storage controller, such as controller 1508, for use with image recognition according to aspects of the present disclosure is shown, wherein any of the aforementioned forms of data augmentation (e.g., skew, rotate, etc.) may be applied. Beginning at block 1702, the data storage controller stores an initial set of labeled (or calibrated) input images within a NAND NVM array of a die coupled to the data storage controller. The input images may be provided by a host device for use with image recognition training (or for use with other forms of deep learning or machine learning). Thereafter, at 1704, when the data storage controller is configured to perform data augmentation (e.g., based on a control signal received from a host), the data storage controller reads one or more of the labeled images from the NAND NVM array. At 1706, the data storage controller generates a set of altered versions of the labeled images by, for example, rotating, translating, skewing, cropping, flipping, and / or adding noise to the labeled images read from the NAND NVM, thereby providing an augmented image set. At 1708, the data storage controller performs machine learning (such as DLA learning) using the augmented image set to, for example, train an image recognition system to recognize images within the augmented image set, and then outputs a set of training parameters to the host (or stores the training parameters in the NAND NVM array). In some examples, the parameters may include synaptic weights, bias values, etc. for use with a DNN configured for image recognition.

[0097] Fig.18 An exemplary method 1800 of data augmentation performed by a data storage controller for use with image recognition according to aspects of the present disclosure is shown, wherein the data augmentation is performed by disabling or at least reducing the use of ECC (or other error correction system or process). Beginning at block 1802, a data storage controller stores an initial set of labeled input images received from a host device within a NAND NVM array of a die. At 1804, the data storage controller disables error detection and correction processes, or otherwise reduces the amount (or effectiveness) of error correction applied to data read from the array, so as to selectively increase the effective BER. At 1806, the data storage controller repeatedly reads images from the NAND NVM array without error correction or with reduced error correction to generate a set of augmented labeled images that differ from each other and from the initial image due to different noise artifacts caused by the lack of error correction or reduced error correction. At 1808, the data storage controller performs machine learning (such as DLA learning) using the augmented labeled image set to, for example, train an image recognition system to recognize images within the set, and then outputs the set of training parameters to the host (or stores the training parameters in the NAND NVM).

[0098] Fig.19An exemplary method 1900 of data augmentation performed by a data storage controller for use with image recognition according to aspects of the present disclosure is shown, wherein data augmentation is performed by modifying NVM read voltages to inject noise into a read image (or otherwise obtain a greater number of read errors). Beginning at block 1902, a data storage controller stores an initial set of labeled input images received from a host device for use within image recognition training within an NVM array of a die. At block 1904, the data storage controller identifies a read voltage for reading data from the NVM array with minimal read errors (e.g., a normal read voltage set to achieve a low BER). At block 1906, the data storage controller modifies the read voltage to be applied to the NVM (compared to a read voltage that would otherwise be used to read an image not subject to data augmentation (e.g., the voltage with minimal read errors identified at block 1904)). At block 1908, the data storage controller controls the read circuitry of the NVM die to apply a modified read voltage to the NVM element while reading one or more initial labeled images from the NVM element (e.g., NAND element) to generate a set of augmented labeled images that differ from each other and from the initial images due to different noise artifacts caused by the modified read voltage. At 1910, the data storage controller performs machine learning (such as DLA learning) using the augmented labeled image set to, for example, train an image recognition system to recognize images within the set, and then outputs the set of training parameters to the host (or stores the parameters in the NVM array).

[0099] Fig. 20 Flow 2000 is generally shown in accordance with some aspects of the present disclosure. Process 2000 may occur within any suitable apparatus or device having a data storage controller capable of performing operations and coupled to an NVM array, such as a NAND die. At block 2002, the data storage controller obtains machine learning data from the NVM array of the memory die. At block 2004, the data storage controller generates augmented machine learning data based on the machine learning data.

[0100] Fig.21 An embodiment of a device 2100 configured in accordance with one or more aspects of the present disclosure is generally shown. The device 2100 or components thereof may be embodied or implemented within an SSD or more generally a DSD having a NAND die or some other type of NVM device supporting data storage. The device 2100 includes a memory die 2012 having an NVM element 2104. The device 2100 also includes a storage controller 2106 configured to obtain machine learning data from the NVM element 2104 of the memory die 2102 and augment the machine learning data.

[0101] Fig. 222200 is shown as an embodiment of an apparatus 2200 configured according to one or more aspects of the present disclosure. Fig.14 As in the example of , the apparatus 2200 or a component thereof may be embodied or implemented within a data storage controller, such as an SSD controller coupled to a NAND die or some other type of NVM device that supports data storage. In various implementations, the apparatus 2200 or a component thereof may be a processor, a controller, a computing device, a personal computer, a portable device, or a workstation, a server, a personal digital assistant, a digital camera, a digital phone, an entertainment device, a medical device, an autonomous vehicle control device, or a component of any other electronic device that stores, processes, or uses neural data.

[0102] The device 2200 includes a communication interface 2202 and is coupled to a NVM 2201 (e.g., a NAND die). The NVM 2201 includes a physical memory array (e.g., a NAND block) 2204 and off-array processing circuitry 2211 (e.g., circuitry below or adjacent to the array). These components may be Fig. 22 The connecting lines in the figure generally indicate that suitable components are coupled to each other and / or placed in electrical communication with each other. Although not shown, other circuits such as timing sources, peripherals, voltage regulators and power management circuits may be provided and will not be described further.

[0103] The communication interface 2202 provides a component for communicating with other devices via a transmission medium. In some specific implementations, the communication interface 2202 includes circuits and / or programming (e.g., programs) suitable for facilitating bidirectional transmission of information relative to one or more devices in the system. In some embodiments, the communication interface 2202 can be configured for wire-based communication. For example, the communication interface 2202 can be a bus interface, a send / receive interface, or some other type of signal interface, including circuits for outputting and / or obtaining signals (e.g., outputting signals from an SSD and / or receiving signals into an SSD).

[0104] The physical memory array 2204 may represent one or more NAND blocks. The physical memory array 2204 may be used to store data, such as images, manipulated by the components of the circuit 2211 and / or the device 2200. The physical memory array 2204 may be coupled to the circuit 2211 so that the components of the circuit 2211 and / or the device 2200 may read or sense information from the physical memory array 2204 and write or program information to the physical memory array. That is, the physical memory array 2204 may be coupled to the components of the circuit 2211 and / or the device 2200 so that the physical memory array 2204 may be accessed by the components of the circuit 2211 and / or the device 2200.

[0105] The device 2200 includes various data augmentation components 2210 that are arranged or configured to acquire, process and / or send data, control data access and storage, issue or respond to commands, and control other desired operations. For example, the components 2210 may be implemented as one or more processors, one or more controllers, and / or other structures configured to perform functions. According to one or more aspects of the present disclosure, the components 2210 may be adapted to perform any or all of the features, processes, functions, operations, and / or routines described herein. For example, the components 2210 may be configured to perform operations relative to Figures 14 to 21 As used herein, the term "adapted to" with respect to component 2210 may refer to the component being configured, employed, implemented and / or programmed to perform one or more of the specific processes, functions, operations and / or routines according to the various features described herein. The circuit may include a special-purpose processor (such as an ASIC) that is used to perform, for example, in conjunction with Figures 14 to 21 Component 2210 is used as an example of a means for processing. In various embodiments, component 2210 may at least partially provide and / or incorporate the above-described Fig.16 The functionality of component 1610 is described.

[0106] According to at least one example of the device 2200, the data expansion component 2210 may include one or more of the following: a circuit / module 2220 configured to store an image or other machine learning data in an NVM; a circuit / module 2222 configured to read an image or other machine learning data from the NVM; a circuit / module 2224 configured to control the expansion of the image or other machine learning data; a circuit / module 2226 configured to skew an image; a circuit / module 2228 configured to crop an image; a circuit / module 2230 configured to flip / rotate / translate an image; a circuit / module 2232 configured to configured for augmentation via noise control; circuit / module 2234 configured for performing ECC; circuit / module 2236 configured for disabling or reducing ECC; circuit / module 2238 configured for adjusting a read voltage of the NVM to inject noise; circuit / module 2239 configured for controlling machine learning, such as controlling DLA, through initial data and augmented data; circuit / module 2241 configured for identifying a worn NVM region; and circuit / module 2243 configured for storing data to and / or reading data from a worn NVM region storage / read component.

[0107] Physical memory array 2204 may include one or more of: blocks 2240 for storing machine learning data; blocks 2242 for storing an expanded version of the machine learning data received from data expansion component 2210; blocks 2244 as wear areas or wear portions; and blocks 2246 for storing other user data or system data (e.g., data related to the overall control of the operation of the NAND die). In some examples, off-array circuitry 2211 of NVM 2201 may include circuits / modules 2213 configured for on-chip data expansion, such as circuits / modules for performing or facilitating data expansion, and / or circuits / modules 2215 configured for ECC. In other examples, NVM 2201 is an otherwise conventional NVM without a data expansion component and / or without on-chip ECC.

[0108] In at least some examples, a method for performing Fig. 22 2220; means for reading the image or other machine learning data from the NVM, such as circuit / module 2222; means for controlling the augmentation of the image or other machine learning data, such as circuit / module 2224; means for skewing the image, such as circuit / module 2226; means for cropping the image, such as circuit / module 2228; means for flipping / rotating / translating the image, such as circuit / module 2230; means for augmentation via noise control, such as circuit / module 2232; means for performing ECC, such as circuit / module 2233; means for performing ECC, such as circuit / module 2234; means for disabling or reducing ECC, such as circuit / module 2235; means for performing ECC, such as circuit / module 2236; means for performing ECC, such as circuit / module 2237; means for performing ECC, such as circuit / module 2238; means for performing ECC, such as circuit / module 2239; means for performing ECC, such as circuit / module 2240; means for performing ECC, such as circuit / module 2241; means for performing ECC, such as circuit / module 2242; means for performing ECC, such as circuit / module 2243; means for performing ECC, such as circuit / module 2244; means for performing ECC, such as circuit / module 2245; means for performing ECC, such as circuit / module 2246; means for performing ECC, such as circuit / module 2247; means for performing ECC, such as circuit / module 2248; means for performing ECC, such as circuit / module 2249; means for performing ECC, such as circuit / module 2250; means for performing ECC, such as circuit / module 2251; means for performing ECC, such as circuit / module 2252; means for performing ECC, such as circuit / module 2253; means for performing ECC, such as circuit / module Module 2236; means for adjusting the read voltage to inject noise, such as circuit / module 2238; means for controlling machine learning with initial data and augmented data, such as circuit / module 2239; means for identifying worn NVM areas, such as circuit / module 2241 or other worn NVM area identifier components; means for storing data to and / or reading data from worn NVM areas, such as circuit / module 2243 or other worn NVM area storage components; means for storing machine learning data, such as NAND block 2240; means for storing an augmented version of the machine learning data, such as NAND block 2242; and means for storing other user data and / or system data (e.g., data relating to the overall control of the operation of the NAND die), such as NAND block 2246.

[0109] Additional aspects

[0110] At least some of the processing circuits described herein may generally be adapted for processing, including the execution of programming code stored on a storage medium. As used herein, the terms "code" or "programming" should be broadly interpreted to include, but are not limited to, instructions, instruction sets, data, code, code segments, program code, programs, programming, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0111] At least some of the processing circuits described herein may be arranged to obtain, process and / or send data, control data access and storage, issue commands, and control other desired operations. In at least one example, the processing circuit may include a circuit configured to implement the desired programming provided by an appropriate medium. For example, the processing circuit may be implemented as one or more processors, one or more controllers, and / or other structures configured to perform executable programming. Examples of processing circuits may include general-purpose processors, digital signal processors (DSPs), ASICs, field programmable gate arrays (FPGAs), or other programmable logic components, discrete gates or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. General-purpose processors may include microprocessors, as well as any conventional processors, controllers, microcontrollers, or state machines. At least some of the processing circuits may also be implemented as a combination of computing components, such as a controller and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with an ASIC and a microprocessor, or a combination of any other number of varying configurations. These examples of processing circuits noted herein are for illustration, and other suitable configurations within the scope of the present disclosure may also be contemplated.

[0112] Aspects of the subject matter described herein may be implemented in any suitable NAND flash memory, such as 3D NAND flash memory. Semiconductor memory devices include volatile memory devices, such as DRAM or SRAM devices, NVM devices, such as ReRAM, EEPROM, flash memory (which may also be considered a subset of EEPROM), ferroelectric random access memory (FRAM) and MRAM, and other semiconductor elements capable of storing information. Each type of memory device may have a different configuration. For example, a flash memory device may be configured in a NAND configuration or a NOR configuration.

[0113] The memory device may be formed of passive elements and / or active elements in any combination. By way of non-limiting example, passive semiconductor memory elements include ReRAM device elements, which in some embodiments include resistivity switching memory elements such as antifuses, phase change materials, etc., and optionally include steering elements such as diodes, etc. By way of further non-limiting example, active semiconductor memory elements include EEPROM and flash memory device elements, which in some embodiments include elements having charge storage regions, such as floating gates, conductive nanoparticles, or charge storage dielectric materials.

[0114] Typically associated circuits are required to operate the memory element and communicate with the memory element. As a non-limiting example, the memory device may have a circuit for controlling and driving the memory element to implement functions such as programming and reading. The associated circuit may be located on the same substrate as the memory element and / or on a separate substrate. For example, a controller for memory read-write operations may be located on a separate controller chip and / or on the same substrate as the memory element. Those skilled in the art will recognize that the subject matter described herein is not limited to the two-dimensional and three-dimensional exemplary structures described, but covers all relevant memory structures within the essence and scope of the subject matter as described herein and as understood by those skilled in the art.

[0115] The examples set forth herein are provided to illustrate certain concepts of the present disclosure. The devices, equipment or components shown above may be configured to perform one or more of the methods, features or steps described herein. Those of ordinary skill in the art will appreciate that these are merely exemplary in nature, and other examples may fall within the scope of the present disclosure and the appended claims. Based on the teachings herein, it will be appreciated by those skilled in the art that the aspects disclosed herein may be implemented independently of any other aspects, and two or more of these aspects may be combined in various ways. For example, any number of aspects set forth herein may be used to implement a device or a method may be practiced. In addition, in addition to or in place of one or more aspects set forth herein, other structures, functions, or structures and functions may be used to implement such a device or such a method may be practiced.

[0116] Various aspects of the present disclosure have been described below with reference to the schematic flow charts and / or schematic block diagrams of the methods, devices, systems, and computer program products according to the embodiments of the present disclosure. It should be understood that each frame of the schematic flow chart and / or schematic block diagram, and the combination of frames in the schematic flow chart and / or schematic block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor or other programmable data processing device of a computer to produce a machine, so that the instructions executed by the processor or other programmable data processing device create a device for implementing the functions and / or actions specified in one or more frames of the schematic flow chart and / or schematic block diagram.

[0117] The subject matter described herein may be implemented in hardware, software, firmware, or any combination thereof. Thus, the terms "function", "module", etc., used herein, may refer to hardware, which may also include software and / or firmware components for implementing the described features. In an exemplary embodiment, the subject matter described herein may be implemented using a computer-readable medium having computer-executable instructions stored thereon, which control the computer to perform the functions described herein when executed by a computer (e.g., a processor). Examples of computer-readable media suitable for implementing the subject matter described herein include non-transitory computer-readable media, such as disk memory devices, chip memory devices, programmable logic devices, and application-specific integrated circuits. In addition, the computer-readable medium implementing the subject matter described herein may be located on a single device or computing platform, or may be distributed across multiple devices or computing platforms.

[0118] It should also be noted that in some alternative embodiments, the functions shown in the frame may not occur in the order shown in the figure. For example, in fact, the two frames shown in succession may be performed substantially simultaneously, or these frames may sometimes be performed in the opposite order, depending on the functions involved. Other steps and methods that are equivalent to one or more frames or parts thereof of the drawings shown in function, logic or effect can be envisioned. Although various arrow types and line types can be used in flow charts and / or block diagrams, it should be understood that these arrow types and line types do not limit the scope of the corresponding embodiments. For example, an arrow can indicate a waiting or monitoring period of unspecified duration between the steps of the enumeration of the depicted embodiment.

[0119] The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, certain methods, events, states or process blocks can be omitted in some specific implementations. The methods and processes described herein are also not limited to any specific sequence, and the blocks or states associated therewith can be performed in other appropriate sequences. For example, the tasks or events described can be performed in a sequence different from that specifically disclosed, or multiple can be combined in a single block or state. Exemplary tasks or events can be performed in series, in parallel or in some other suitable manner. Tasks or events can be added to or deleted from the disclosed exemplary embodiments. The exemplary systems and components described herein can be configured differently from those described. For example, compared with the disclosed exemplary embodiments, elements can be added, removed or rearranged.

[0120] Those skilled in the art will appreciate that any of a variety of different techniques and technologies may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above specification may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0121] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects. Likewise, the term "aspects" does not require that all aspects include the discussed feature, advantage, or mode of operation.

[0122] Although the above description includes many specific embodiments of the present invention, these should not be interpreted as limiting the scope of the invention, but as examples of its specific embodiments. Therefore, the scope of the present invention should not be determined by the illustrated embodiments, but should be determined by the attached claims and their equivalents. In addition, "one embodiment", "embodiment" or similar language mentioned throughout this specification refers to the specific features, structures or characteristics described in conjunction with the embodiment being included in at least one embodiment of the present disclosure. Therefore, the phrases "in one embodiment", "in embodiments" and similar language that appear throughout this specification may but do not necessarily all refer to the same embodiment, but refer to "one or more but not all embodiments", unless otherwise explicitly stated.

[0123] The terms used herein are only for the purpose of describing specific aspects and are not intended to limit these aspects. As used herein, the singular forms "one", "an" and "the" are also intended to include plural forms (i.e., one or more), unless the context clearly indicates otherwise. The enumerated list of items does not mean that any or all of the items in the project are mutually exclusive and / or mutually inclusive, unless otherwise clearly stated. It will be further understood that, unless otherwise clearly stated, the terms "comprises", "comprising", "includes", "including", "having", and its variants used in this article represent "including but not limited to". That is, these terms can specify the existence of stated features, integers, steps, operations, elements or parts, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, parts or their groups. In addition, it should be understood that the word "or" has the same meaning as the Boolean operator "OR", that is, it includes the possibility of "or" and "both", and is not limited to "exclusive or" ("XOR"), unless otherwise clearly stated. It should also be understood that the symbol " / " between two adjacent words has the same meaning as "or", unless otherwise explicitly stated. In addition, phrases such as "connected to", "coupled to" or "in communication with" are not limited to direct connection, unless otherwise explicitly stated.

[0124] Any reference to an element using names such as "first", "second" etc. herein does not generally limit the quantity or order of those elements. On the contrary, these designations can be used as a convenient method to distinguish two or more elements or element instances in this article. Therefore, the reference to the first and second elements does not mean that only two elements can be used there, or that the first element must be before the second element in some way. In addition, unless otherwise specified, a group of elements may include one or more elements. In addition, the term "at least one of A, B or C" or "A, B, C or any combination thereof" used in the specification or claims refers to "A or B or C or any combination of these elements". For example, this term can include A, or B, or C, or A and B, or A and C, or A and B and C, or 2A, or 2B, or 2C, or 2A and B, etc. As another example, "at least one of A, B or C" is intended to cover A, B, C, AB, AC, BC and ABC, and multiples of the same members (for example, including any list of AA, BB or CC). Likewise, "at least one of A, B, and C" is intended to encompass A, B, C, AB, AC, BC, and ABC, and multiples of the same member. Similarly, as used herein, a phrase referring to a list of items linked with "and / or" refers to any combination of items. As an example, "A and / or B" is intended to encompass A alone, B alone, or A and B together. As another example, "A, B, and / or C" is intended to encompass A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0125] As used herein, the term "determining" encompasses various actions. For example, "determining" may include arithmetic, calculation, processing, derivation, investigation, lookup (e.g., lookup in a table, database, or another data structure), determination, etc. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Also, "determining" may include parsing, selecting, choosing, establishing, etc.

Claims

1. A device comprising: a die having a non-volatile memory (NVM) element; and a data augmentation controller formed in the die and configured to augment the machine learning data stored in the NVM element by augmenting the machine learning data; wherein the apparatus further comprises a read component configured to apply a read voltage to the NVM element to read data from the NVM element, and wherein the data expansion controller is further configured to generate the expanded data by modifying the read voltage to maintain a larger amount of read errors than would occur when reading data not subject to data expansion, and then applying the modified read voltage to the NVM element while reading stored data from the NVM element.

2. The apparatus of claim 1 , wherein the machine learning data comprises images, and the data augmentation controller is further configured to generate augmented images based on one or more initial images obtained from the NVM element.

3. The apparatus of claim 2 , wherein the data augmentation controller is further configured to generate the augmented image by applying one or more of rotation, translation, skew, cropping, flipping, and noise addition to the one or more initial images obtained from the NVM element to generate one or more altered images.

4. The device according to claim 1, wherein the apparatus further comprises an error correction component configured to correct errors in data read from the NVM element, and Wherein the data expansion controller is further configured to generate the expansion data by disabling the error correction component and then reading the stored data from the NVM element without error correction.

5. The device according to claim 1, wherein the apparatus further comprises an error correction component configured to correct errors in data read from the NVM element, and wherein the data expansion controller is further configured to generate the expanded data by reducing an amount of error correction performed by the error correction component compared to an amount of error correction that would otherwise be used to read data not subject to data expansion, and then reading the stored data from the NVM element with the reduced error correction.

6. The apparatus according to claim 1, wherein the data expansion controller further comprises: a worn NVM region identifier component configured to identify a worn region of the NVM element that is subject to a storage error; and A wear region storage component is configured to store an un-augmented version of data in the wear region of the NVM element and then read the data from the wear region of the NVM element to obtain a noisy augmented version of the data.

7. The apparatus of claim 1, wherein the NVM element comprises a NAND flash memory element.

8. The apparatus of claim 1, wherein the data expansion controller formed in the die is further configured as one or more of an under-array component and an array-adjacent component.

9. The apparatus of claim 1 , further comprising a machine learning component formed in the die and configured to perform machine learning using the augmented machine learning data.

10. A method for use with a die having a non-volatile memory (NVM) array, the method comprising: storing machine learning data in the NVM array of the die; as well as generating augmented machine learning data using data augmentation circuitry formed in the die; Generating the expanded machine learning data further includes: modifying a read voltage applied to the NVM element as compared to a read voltage that would otherwise be employed by a read component to read an image not subject to data expansion; as well as The modified read voltage is applied to the NVM element while simultaneously reading machine learning data from the NVM element.

11. The method of claim 10 further comprising using a machine learning controller formed in the die to perform at least one machine learning operation using the augmented machine learning data.

12. The method of claim 11, wherein performing the at least one machine learning operation further comprises training a deep learning accelerator (DLA) using the augmented machine learning data.

13. The method of claim 10, wherein the machine learning data comprises labeled images, and the augmenting the machine learning data comprises augmenting the labeled images.

14. The method of claim 13, wherein generating the augmented machine learning data further comprises: reading one or more of the tag images from the NVM array; as well as One or more of rotating, translating, skewing, cropping, flipping and adding noise are performed to one or more of the labeled images to generate one or more altered images.

15. The method of claim 10, wherein generating the augmented machine learning data further comprises: reducing usage of an error correction component configured to apply error correction to data read from the NVM array; as well as Machine learning data is read from the NVM array with reduced error correction.

16. The method of claim 10, wherein generating the augmented machine learning data further comprises: identifying a portion of the NVM array that is subject to a storage error; reading machine learning data from a first portion of the NVM array; storing the machine learning data in the portion of the NVM array that was subject to the storage error; as well as The machine learning data is read from the portion of the NVM array that is subject to storage errors to obtain a noise-augmented version of the machine learning data.

17. A data storage device, the data storage device comprising: a memory die having a non-volatile memory (NVM) element; and A storage controller, wherein the storage controller is configured to: obtaining machine learning data from the NVM elements of the memory die and augmenting the machine learning data; wherein the memory die includes a read component configured to apply a read voltage to the NVM element to read data from the NVM element, and wherein the memory controller is further configured to generate the expanded data by controlling the memory die to modify the read voltage to maintain a larger amount of read errors than would occur when reading data not subject to data expansion, and then controlling the memory die to apply the modified read voltage to the NVM element while reading stored data from the NVM element.

18. The data storage device of claim 17, wherein the machine learning data comprises an image, and the storage controller is further configured to generate an augmented image based on one or more initial images obtained from the NVM element.

19. The data storage device of claim 18, wherein the storage controller is further configured to generate the augmented image by applying one or more of rotation, translation, skew, cropping, flipping, and noise addition to the one or more initial images obtained from the NVM element to generate one or more altered images.

20. The data storage device according to claim 17, wherein the memory controller includes an error correction component configured to correct errors in data read from the NVM element of the memory die, and wherein the memory controller is further configured to generate augmented data by controlling the error correction component to increase the amount of noise in the data read from the NVM element.

21. A data storage device according to claim 20, wherein the storage controller is further configured to generate the expanded data by reducing the amount of error correction performed by the error correction component compared to the amount of error correction that would otherwise be used to read data not subject to data expansion, and then reading the stored data from the NVM element of the memory tube core with the reduced error correction.

22. The data storage device of claim 17, wherein the storage controller further comprises: a worn NVM region identifier component configured to identify a worn region of the NVM element of the memory die that is subject to a memory error; and A wear area storage component is configured to control the memory die to store an un-expanded version of data in the wear area of ​​the NVM element of the memory die and then read the data from the wear area of ​​the NVM element to obtain a noisy expanded version of the data.

23. The data storage device of claim 17, wherein the NVM elements of the memory die comprise NAND storage elements.

24. The data storage device of claim 17, further comprising a plurality of memory dies, and wherein the memory controller is further configured to: obtaining machine learning data from the NVM elements of the plurality of memory dies in parallel; and The machine learning data obtained from the plurality of memory dies is augmented.

25. A data storage device according to claim 24, wherein the storage controller is further configured to control the multiple memory tube cores to adjust one or more of error correction within the memory tube core, read voltage within the memory tube core, and storage of an unexpanded version of data in the wear area of ​​the NVM element of the memory tube core.

26. A method for use by a storage controller of a data storage device, the method comprising: Obtaining machine learning data from a non-volatile memory (NVM) array of a memory die; as well as generating, at the storage controller, augmented machine learning data based on the machine learning data; Wherein using the storage controller to generate the augmented machine learning data further comprises: controlling the memory die to modify a read voltage applied to the NVM element as compared to a read voltage that would otherwise be employed by a read component to read an image not subject to data expansion; as well as The memory die is controlled to apply the modified read voltage to the NVM element while reading machine learning data from the NVM element.

27. The method of claim 26, further comprising performing at least one machine learning operation using the storage controller using the augmented machine learning data.

28. The method of claim 27, wherein using the storage controller to perform the at least one machine learning operation further comprises using the augmented machine learning data to train a deep learning accelerator (DLA).

29. The method of claim 26, wherein the machine learning data comprises labeled images, and wherein augmenting the machine learning data comprises augmenting the labeled images.

30. The method of claim 29, wherein using the storage controller to generate the augmented machine learning data further comprises: reading one or more of the tag images from the NVM array; as well as One or more of rotating, translating, skewing, cropping, flipping and adding noise are performed to one or more of the labeled images to generate one or more altered images.

31. The method of claim 26, wherein using the storage controller to generate the augmented machine learning data further comprises: reducing usage of an error correction component configured to apply error correction to data read from the NVM array; as well as Machine learning data is read from the NVM array with reduced error correction.

32. The method of claim 26, wherein using the storage controller to generate the augmented machine learning data further comprises: identifying a portion of the NVM array that is subject to a storage error; controlling the memory die to store the machine learning data in the portion of the NVM array that was subject to a storage error; as well as The machine learning data is read from the portion of the NVM array that is subject to storage errors to obtain a noise-augmented version of the machine learning data.

33. The method of claim 26, wherein the data storage device comprises a plurality of memory dies, and wherein the method further comprises: obtaining machine learning data from the NVM elements of the plurality of memory dies in parallel; as well as The machine learning data obtained from the plurality of memory dies is augmented in parallel with augmenting the machine learning data.

34. An apparatus for use with a data storage device, the apparatus comprising: means within a storage controller of the data storage device for obtaining machine learning data from a non-volatile memory (NVM) array of a memory die of the data storage device; and means within the storage controller of the data storage device for generating augmented machine learning data from the machine learning data; Generating the expanded machine learning data further includes: controlling the memory die to modify a read voltage applied to the NVM element as compared to a read voltage that would otherwise be employed by a read component to read an image not subject to data expansion; as well as The memory die is controlled to apply the modified read voltage to the NVM element while simultaneously reading machine learning data from the NVM element.

Images