Storage device with hardware accelerator

Abstract

The storage device includes a storage controller, a flash memory, and a hardware accelerator communicatively coupled. The hardware accelerator is configured to selectively retrieve data stored in the flash memory in response to a request for the data, and can perform other operations to accelerate data access of the computer system.

Description

Technical Field

[0001] Various aspects of this disclosure relate to storage devices. In particular, various aspects of this disclosure relate to flash memory devices that include hardware accelerators. Background Technology

[0002] Solid-state storage devices, such as flash memory, are rapidly becoming the most popular type of digital storage device in computer applications. The use of flash memory devices (also known as thumb drives) based on the Universal Serial Bus (USB) has reached near ubiquitous use. The USB protocol has been updated to improve transfer speeds, allowing for a wider range of storage usage in computer systems. Another protocol, Fast Peripheral Component Interconnect (PCIe), is also prevalent for high-speed interconnects between devices and computer systems. PCIe provides high-speed connections to computer systems, even offering some hot-swapping capabilities, but has long been used for permanently connecting certain types of devices, such as graphics cards and hard drives, to computer systems. Unlike USB, the PCIe hardware interface uses a large-edge double-sided connector with pin assignments for full-size cards up to 49 pins, and the smallest double-sided connector has 18 pins. USB-C, on the other hand, uses a 12-pin connector. A newer PCI interconnect standard, M.2, has also recently gained widespread adoption. M.2 provides a 59-pin high-speed physical interface with a smaller form factor for PCIe. Furthermore, the new protocol for the Non-Volatile Memory Express (NVMe) interface specification for solid-state storage devices has been widely adopted. NVMe allows for greater parallelism in communication with solid-state storage devices, thus providing higher transfer speeds. NVMe also allows for additional functionality for connected devices, such as the NVMe over Fabrics (NVMe-OF) specification, which allows connections to devices using transport protocols such as TCP, which connect to the NVMe physical interface of a computer system. These new protocols and standards pave the way for new capabilities that can be implemented on storage devices, improving the functionality of the attached computer systems. Summary of the Invention

[0003] It is against this backdrop that the various aspects of this disclosure have emerged. Attached Figure Description

[0004] The teachings of this disclosure can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, wherein:

[0005] Figure 1 A system block diagram depicts a computer system connected to an improved storage device according to aspects of this disclosure.

[0006] Figure 2 This is a device block diagram depicting an implementation of an improved storage device with a GPU accelerator according to one aspect of this disclosure.

[0007] Figure 3 This is an apparatus block diagram depicting an implementation of an improved storage device with an FPGA-based accelerator according to one aspect of this disclosure.

[0008] Figure 4 This is an apparatus block diagram depicting an embodiment of an improved storage device with a neural network processor according to one aspect of the present disclosure.

[0009] Figure 5 This is a block diagram illustrating an improved storage device that communicates over a network via connected computer systems.

[0010] Figure 6 This is a timing diagram of the operation of an improved storage device with a hardware accelerator according to aspects of this disclosure.

[0011] Figure 7 It is a timing diagram of the operation of an improved storage device with a GPU accelerator according to various aspects of this disclosure.

[0012] Figure 8 This is a timing diagram of the operation of an improved storage device with an NPU according to various aspects of this disclosure. Detailed Implementation

[0013] Although the following detailed description contains many specific details for illustrative purposes, those skilled in the art will understand that many variations and modifications to these details are within the scope of the invention. Therefore, examples of embodiments of the invention described below are set forth without losing the generality of the claimed invention or imposing limitations on it.

[0014] According to aspects of this disclosure, storage devices can improve the functionality of connected computer systems by adding hardware accelerators. Figure 1A system block diagram depicts a computer system 102 connected to an improved storage device according to aspects of this disclosure. As shown, the improved storage device 101 may include flash memory 104, a storage controller 106, and a hardware accelerator 103. In some alternative embodiments, the storage device may also include a central processing unit (CPU), and in some embodiments, may also include random access memory (RAM) for the CPU 107. The RAM may include a storage device-specific operating system (OS). By way of example and not limitation, in some embodiments, the hardware accelerator 103, flash memory 104, and storage controller 106 may be located on the same circuit board. In alternative embodiments, the optional central processing unit and memory 107 may be located on the same circuit board as the storage controller, flash memory, and hardware accelerator.

[0015] The improved storage device 101 can communicate with the computer system 102 via hardware interface 105. The computer system 102 may include a storage device host controller 108. The storage device host controller 108 may include a control block for the storage device as I / O device 110, and optionally, a control block for the storage device as network device 109. The I / O device control block 110 may allow the storage device 101 to communicate with the file system 115 of the computer system 102. In some alternative embodiments, the network device control block 109 may communicate with the hardware accelerator 103 in the improved storage device 101. The network device control block 109 may present the improved storage device as a device connected to the computer system via network connection 113, rather than via hardware interface 105, and may communicate with the file system of the computer system 115 as an analog standard network interface. Additionally, the network device control block 109 may allow the improved storage device 101 to communicate over network connection 113 via network bridge 111. Network bridge 111 bridges the analog standard network interface within the computer system's internal hardware with an external network connection 113. This bridging between the network and the enhanced storage device allows the hardware accelerator to connect to a network, such as the Internet, using the computer hardware's built-in routing. This can allow other computers to access other components of the hardware accelerator and / or the enhanced storage device, while protecting the computer system and any separate internal or private networks it may be connected to.

[0016] As an example, in some implementations, routing rules can be configured to allow accelerator 103 to access a wide area network such as the Internet while blocking access to the private network of computer system 102. Routing rules can act as an equivalent of a Virtual Private Network (VPN), which isolates traffic from accelerator 103 to the Internet from internal traffic of computer system 102. Both the computer system and the accelerator can communicate with the Internet, but outgoing traffic cannot access the separate network of the computer system. This is an important feature in implementations where it is desirable or even necessary to allow computer system 102 to provide inbound traffic to accelerator 103 and for the accelerator to respond to such traffic, but it is also desirable or necessary to prevent the accelerator from monitoring or initiating connections to the separate network of the computer system.

[0017] The computer system may include computer hardware components 112, such as a CPU, memory, local storage device, or GPU. The computer hardware components 112 may operate together to perform computer functions and may communicate with the enhanced storage device during operation. For example, but not limited to, the computer hardware components may receive data from the enhanced storage device, send requests for data to the enhanced storage device, send storage requests to the enhanced storage device, and send data to be stored in the enhanced storage device. Additionally, in some embodiments, the computer hardware components may operate with a storage device host controller 108 to allow messages from a network connection to reach the enhanced storage device 101.

[0018] An improved feature of storage device 101 is hardware accelerator 103. Hardware accelerator 103 can be configured to offload some conventional processes performed by computer hardware component 112 to the hardware accelerator. Hardware accelerator 103 may include application-specific integrated circuits (ICs), such as graphics processing unit accelerator (GPU) ICs, neural network processor (NPU) ICs, or field-programmable gate array (FPGA) ICs. For example, but not limited to, hardware accelerator 103 can index data from computer system 102 stored in flash memory 104 by generating index entries for data from computer system stored in flash memory 104, thereby improving data access time of computer system and reducing the amount of data that needs to be transferred between flash memory and computer system 102. In addition, hardware accelerator 103 can perform other functions, such as data encryption, data description, data compression, data decompression, etc. Data compression may include a corresponding process of encoding data, and data decompression may include a corresponding process of decoding data. Some examples of compression algorithms that can be used by the hardware accelerator include, but are not limited to, Lempel-Ziv-Welch (LZW), entropy coding algorithms, run-length encoding, etc. Furthermore, storage devices with hardware accelerators can read data provided to them by a computer system and select how to process the data based on its data type. The data type can include file type, data arrangement, data size, etc. For example, but not limited to, the improved storage device can use a first index to store image data and a second index to store audio data, thereby reducing the search time for both audio and image data. In some alternative implementations, the improved storage system may be able to read and search data within a data array; for example, but not limited to, the improved storage device may be able to search the data array, such as a row-based data array file or a column-based data array file, to find data queries sent to the hardware device by the computer system. An example of a column-based array file type that can be used in some embodiments of this disclosure is the Apache Parquet format file type. It should be understood that the array can be an implementation of vector-oriented storage. Vector-oriented storage can use one or more machine learning models to reduce the dimensionality and tokenize the data, resulting in data called embeddings. Embeddings can be stored in flash memory with an index referencing the location of the original data. Therefore, vector-oriented databases can be dedicated to Single Instruction Multiple Data (SIMD) architectures used in GPUs and some FPGAs, which may be more efficient in machine learning tasks. Therefore, in some vector-based file types, the block size for different parts is fixed, so hardware accelerators can be programmatically determined to point each core to scan data independently without prior knowledge of other blocks.In some alternative implementations, the data in the flash memory can be assets such as videos, images, audio, game scripts, etc., and the hardware accelerator can create derivative assets from the underlying assets stored in the flash memory 104.

[0019] Figure 2 This is a device block diagram depicting an embodiment of an improved storage device with a GPU accelerator according to one aspect of this disclosure. In this embodiment, the improved storage device 200 includes a GPU-type hardware accelerator 201, flash memory 202, a storage controller 203, and a hardware interface 204, which are communicatively coupled via a main data bus 204. Optionally, the improved storage device may include a CPU 207, which may also be communicatively coupled to random access memory (RAM) 208. The optional CPU and RAM may be integrated together in a single system-on-a-chip (SoC) package. RAM 208 may include instructions for operation of a storage device-specific operating system implemented by CPU 207. The optional CPU 207 may be any suitable CPU. Alternatively, instructions for a storage device-specific OS may be stored in a portion of the flash memory 202.

[0020] The GPU-type hardware accelerator 201, referred to below as a GPU accelerator, can perform functions to assist the operation of the GPU in a computer system. For example, but not limited to, the GPU accelerator 201 can use indexes to perform data storage and retrieval. In some embodiments, the stored data can be texture data indexed by at least a Level of Detail (LOD); the hardware accelerator can be configured to retrieve the correct texture data based on the LOD provided by the computer system. In some embodiments, the texture data indexed by at least a LOD can be a MIP map stored in flash memory, and the GPU accelerator can retrieve the correct resolution texture based on a request from the computer system. In some specific embodiments, the GPU accelerator can also perform other functional graphics processing functions, such as graphics pipeline operations. For example, but not limited to, the GPU accelerator can be configured to perform shading and filtering operations on images in flash memory. According to one aspect of this disclosure, the GPU accelerator can be configured to allow a dedicated language similar to OpenGL to operate the hardware accelerator and perform tasks. This can provide software developers with the ability to define how to efficiently manage data stored in flash memory. One aspect of some embodiments of this disclosure is that the GPU accelerator can see and use flash memory on the improved storage device in the same way that a GPU on a computer can access memory (sometimes called VRAM) located in the GPU. In some other alternative embodiments, the GPU accelerator may include hardware codecs and / or may be configured to perform video encoding and decoding. Additionally, the GPU accelerator may be configured to perform heavy data manipulation and filtering. In some embodiments, the GPU accelerator may be configured to implement one or more trained machine learning models. The GPU accelerator, along with other components of the storage device, may include a flexible, open storage language that allows custom data filters to be generated by the user and executed using the GPU accelerator. The GPU accelerator may be configured to execute custom filters that manipulate arrays of data stored in flash memory and provide the manipulated data to the computer system, thereby reducing the processing workload of the computer system. Filters executed by the GPU accelerator may perform multiplication, addition, subtraction, division, removal, copying, formatting, algorithmic operations, etc., on the data stored in flash memory. In some implementations, the computer system can request specific data, and the GPU accelerator can filter arrays stored in flash memory to obtain specific data, thereby reducing the amount of data transmitted to the computer system and eliminating the need for the computer system itself to filter data.

[0021] Storage controller 203 can be any suitable storage controller in the art, such as, but not limited to, an NVMe controller, a PCIe controller, a USB controller, etc. Similarly, hardware interface 205 can be any hardware interface compatible with the storage controller. Suitable hardware interfaces can be, for example, but not limited to, a USB connector, an M.2 connector, a PCIe edge connector, etc. Flash memory can be any suitable solid-state storage device.

[0022] Figure 3This is a device block diagram depicting an alternative embodiment of an improved storage device 300 with an FPGA-based accelerator according to one aspect of this disclosure. The FPGA accelerator 301 can perform similar functions to the GPU-type accelerator 201, but can be more specialized for purposes such as database manipulation. The FPGA-based accelerator 301 can be any suitable field-programmable gate array known in the art, and in some embodiments can be a custom integrated circuit for the improved storage device. The FPGA accelerator 301 can perform data storage with flash memory using indexes and retrieval using the indexes. Index entries can be generated by the FPGA accelerator. Additionally, the FPGA accelerator 301 can be configured to perform data remanipulation and filtering on data stored in flash memory 302. The FPGA accelerator, along with other components of the storage device, can be loaded with a flexible open storage language that allows users to generate custom data filters and execute them using the FPGA accelerator. The FPGA accelerator can be configured to execute custom filters that manipulate data in a database stored in flash memory 302 and provide the manipulated data to a computer system, thereby reducing the processing workload of the computer system. The data in the database can be arranged in an array, such as a row-based array or a column-based array format. As mentioned above, the database can be a vector-oriented database. Filters executed by FPGA accelerators can perform multiplication, addition, subtraction, division, removal, copying, formatting, and algebraic operations on data stored in flash memory. In some implementations, a computer system can request specific data, and the FPGA can filter the array stored in flash memory to obtain that specific data, thereby reducing the amount of data transmitted to the computer system and eliminating the need for the computer system itself to filter the data. FPGAs can be configured to include codecs implemented in dedicated hardware (also known as HardIP codecs) or codecs implemented in software running on more general-purpose hardware (also known as SoftIP codecs), which can perform encoding and decoding functions such as video encoding, encryption, hashing, etc. In some implementations, FPGAs can perform codec functions on a combination of SoftIP and HardIP. FPGAs can include SoftIP implementing Video Hardware Description Language (VHDL) blocks and HardIP blocks, such as memory controllers or PCIe controllers, which are more efficient in processing operations than SoftIP because HardIP is configured on a silicon substrate and does not consume the limited resources of FPGA logic units (LUTs).

[0023] Figure 4This is a device block diagram depicting an alternative embodiment of an improved storage device with a neural network processor according to one aspect of the present disclosure. In this embodiment, the storage device includes an NPU 401 communicatively coupled to flash memory 402. The NPU 401 may be an application-specific integrated circuit (ASIC) dedicated to performing machine learning functions. In some embodiments, the NPU 401 may include a pre-trained neural network model formed in hardware. In some embodiments, the NPU 401 may include one or more floating-point computation subunits configured to rapidly perform mathematical operations associated with machine learning.

[0024] According to one aspect of this disclosure, NPU 401 can read flash memory 402 and is configured to generate derived assets from underlying assets stored in the flash memory. NPU 401 can load neural network (NN) data, such as an NN model with corresponding weights, transition values, etc. The NN data enables the NPU to implement a pre-trained neural network model for asset generation; for example, but not limited to, the NPU can implement generative pre-trained neural networks, such as autoencoder-type models, diffusion-type models, etc. Examples of autoencoder-type models include transformer-type models, such as chat generation pre-trained transformers (GPT), bidirectional encoder representations from transformers (BeRT), language models for conversational applications (LaMDA), etc. In some implementations, the NPU can implement other neural network models. For example, but not limited to, other neural network models can be neural radiation field (NeRF) models. Various aspects of this disclosure additionally implement a custom language that allows developers to provide custom instructions to the hardware controller for machine learning models. Therefore, there is no limitation on the machine learning models that can be implemented by improved storage devices. Furthermore, aspects of this disclosure are not limited to generating machine learning models or unsupervised machine learning models, but can also be applied to other ML models such as supervised learning models, where the model is trained and frozen under the supervision of the developer, and then the trained model is used without further tuning. For example, but not limited to, improved storage devices can enable developers to write custom scripts to represent inference models and scan data in flash memory using hyperparameters from training. Inference actions can take the form of data requests from a computer to a hardware accelerator to "query" data using hyperparameters as filtering expressions with the inference model.

[0025] The NPU 401, implementing a neural network model, can be configured to generate one, two, or more derived assets using base asset data 403 stored in flash memory. For example, but not limited to, the derived assets can be event scripts or character dialogues, the neural network model can be a pre-trained generative model (such as a large language model like GPT) trained with machine learning algorithms to generate event scripts and / or character dialogues, and the base asset can be a cue for generating specific events and / or character dialogues. When the computer system invokes the base asset using a generation command, the NPU can generate the event script or character dialogue based on the cue. This allows the computer system to store only a small amount of data for cue in flash memory and generate a large amount of data for scripts and / or character dialogues.

[0026] In another example, the neural network model could be a pre-trained generative model for images, and the underlying assets could be images, videos, one or more frames from a video, or text prompts stored in flash memory.

[0027] When a computer system requests a base asset with a generate command, the NPU can use a pre-trained machine learning model (such as a diffusion model) trained with machine learning algorithms to generate various derived images, videos, frames from videos, or models (depending on the request and the type of machine learning model) from the base asset. In another implementation, the NPU can utilize machine learning algorithms to implement a deep learning neural network model, such as a NeRF model, and the base asset comprises two or more image views of a scene or object. When the computer requests the base asset with a generate command, the NPU can generate a 3D representation of the base asset.

[0028] Figure 5This is a block diagram illustrating an improved storage device communicating over a network via connected computer systems. In the illustrated embodiment, a first improved storage device 501 is connected to a first computer system 503. The first computer system 503 is communicatively coupled to a second computer system 504 via a network 505. A second improved storage device 502 is connected to the second computer system 504 via a hardware interface. The first improved storage device 501 emulates a standard network interface 506 on the first computer system 503, thereby allowing the first improved storage device 501 to send and / or receive data with the second computer system 504 via the network 505. Additionally, the second improved storage device 502 emulates a standard network interface 507, thereby allowing the second computer system 504 to send and / or receive data to the second improved storage device 503 using the emulated standard network interface. The first computer system 503 can send and / or receive data from the second improved storage device 502. This configuration can allow a remote computer system to access and / or modify files on the improved storage device. For example, a user of the second computer system can create a file filter and upload that filter to their improved storage device. The improved storage device can have a privileged system that allows users to select who can write, read, or modify files on the improved storage device. A user of the first computer system can have privileges set to allow other computer systems to read their filter files on the improved storage device. Therefore, the first user can access the second improved storage device and download their filters without accessing the file system of the second user's computer system. In another example, the improved storage device can allow coordinating multiple improved storage devices to provide increased computing power for computational tasks. In this embodiment, the second computer system can set privileges on the second improved storage device to allow the first computer system to read, write, and modify files on the second improved storage device. The first computer system can send a portion of data to the second improved storage device and the first storage device, requesting computational operations to be performed on the corresponding data portion. In response, the first and second improved storage devices can execute the request and send the results back to the first computer system, where the resulting data portion can be assembled. In some embodiments, the emulated standard network interface can appear to the computer system's operating system as a wireless or wired networking interface, or a network-attached storage device, and therefore can be compatible with the computer system's local networking system and can be controlled by standard computer networking mechanisms. For example, but not limited to, network firewall rules can be defined to restrict access, ensuring that the improved storage device can only communicate with the Internet and cannot access the computer's file system, thereby providing isolation. In some other implementations, network access to the improved storage device can be controlled by network components (such as, but not limited to, network switches) or by the improved storage device itself.The improved storage device can provide a virtual interface that allows users to set network access rules and / or user permissions.

[0029] Figure 6 This is a timing diagram illustrating the operation of an improved storage device with a hardware accelerator according to various aspects of the present disclosure. As shown, computer system 601 communicates with improved storage device 620. Computer system 601 can send a request for stored data 604 to hardware accelerator 602. This request can be relayed to the hardware accelerator via the storage controller. In an exemplary embodiment according to aspects of the present disclosure, accelerated reads can be sent via an emulated network interface on the hardware accelerator, which presents the memory as network-attached storage. In this embodiment, write requests to the improved storage device can be executed via the block I / O interface of the storage controller, which represents the storage as a system file system. Alternatively, both reads and writes can occur via the block I / O interface, but reads may not be accelerated due to the low-level characteristics of the storage controller's block I / O interface.

[0030] According to another aspect of this disclosure, the improved storage controller may include a system-on-a-chip (SoC) that includes a linked CPU and can be configured to treat data on flash memory as a native file system of the SoC. Thus, the hardware accelerator and the SoC can "mount" flash memory as an attached storage drive to the SoC, and accelerated reads from the network interface can then perform special file operations at the file system level, not limited to reads and writes at the raw memory block level. Therefore, the improved storage device can be referred to as file-aware because the SoC implements file system-level operations on data stored in flash memory. Additionally, any proprietary data format can still reside on the low-level formats EXT5 / ZFS / NTFS provided through the storage controller's data interface. The SoC can read files like an attached drive and then use the file system as a bootstrap for the raw data from the perspective of raw memory. The file system on the improved storage device can be augmented with indexed data, so when the SoC opens a file, it can index it, eliminating the need to scan the entire flash memory to determine where the actual data blocks reside, for example, after directory / node traversal occurs. In some implementations, an index may not be necessary to support custom data formats, but it can be used for a "cached" file storage layout, allowing the hardware accelerator to allocate cores to memory ranges. In some implementations, read requests to the hardware accelerator can use filenames, and the hardware accelerator can locate an index on a memory segment containing the desired file, then return the result to the file read request after scanning the location of the desired file in flash memory. On-SoC O / S can allow the device to have native support for file systems or standard libraries used (e.g., but not limited to, the Java Parquet file reader).

[0031] Hardware accelerator 602 can read data and organize it according to indexes. For example, in the case of data corresponding to textures with different Levels of Degrees (LODs), each index can correspond to a different LOD. The hardware accelerator can then send a write command 605 to flash memory 603 according to the index. In some embodiments, the write command 605 can be relayed through a storage controller. Additionally, in this embodiment, hardware accelerator 602 also sends write requests to entries in index 606 of the data stored in flash memory 603.

[0032] In some implementations, the indexing of data can be file- and / or data-type selective. In such implementations, the hardware controller can read data or metadata to determine the file type and / or data type of the data, and send an indexed data storage request based on the determined file type and / or data type. To perform data indexing, in some implementations, the hardware accelerator can maintain an internal register representation of the index stored in flash memory. Alternatively, the hardware accelerator can read the index from flash memory before organizing the data according to the index.

[0033] Computer system 601 may request to read data 607 from improved storage device 602. This read request 607 may be passed to hardware accelerator 602. In some embodiments, the read request may initially be received by the storage controller before being passed to hardware accelerator 602. Upon receiving the read request, the hardware accelerator may selectively send a read request to flash memory 603 for index data corresponding to the index of the data 608 to be read. Selectively sending the read request may include reading data to determine whether the file type and / or data type of the data to be read is stored with the index, or, if multiple indexes exist for different data types or file types, which index should be read to determine the location of the data to be read. Flash memory 603 sends index data, including the index location 609 of the data to be read, back to hardware accelerator 602. Hardware accelerator 602 then uses the index location to send a read request 610 to flash memory 603 for the data at the location indicated by the index. In response, flash memory 603 may send the requested index data 611 back to the hardware accelerator. Alternatively, flash memory 603 may instead send the requested index data to the storage controller. Once data is received at the hardware accelerator 602 or the storage controller, it can be sent 612 to the computer system 601.

[0034] Figure 7This is a timing diagram illustrating the operation of an improved storage device with a GPU accelerator according to various aspects of this disclosure. As shown, computer system 701 communicates with improved storage device 720. Computer system 701 may send a request to GPU accelerator 702 to store texture data 704. This request may be relayed to GPU accelerator via storage controller. GPU accelerator 702 may read the texture data and organize the texture data according to Level of Detail (LOD). GPU accelerator may then send a write command for texture data 705 to flash memory 703 according to LOD. In some embodiments, write command 705 may be relayed via storage controller. Additionally, in this embodiment, GPU accelerator 702 also sends a write request to an entry in the LOD index 706 of the texture data stored in flash memory 703. In order to perform indexing of texture data by LOD, in some embodiments, hardware accelerator may maintain an internal register representation of the LOD index stored in flash memory. Alternatively, hardware accelerator may read the LOD index from flash memory before organizing the data according to LOD.

[0035] Computer system 701 may request to read texture data at a specific LOD 707 from improved storage device 702. This read request 707 may be passed to GPU accelerator 702. In some embodiments, the read request may initially be received by the storage controller before being passed to GPU accelerator 702. After receiving the read request, GPU accelerator may selectively send the read request for texture data indexed by LOD 708 to flash memory 703. In some embodiments, GPU accelerator may examine the request to determine if it is for texture data. Flash memory 703 sends texture data 709 back to GPU accelerator 702. Alternatively, flash memory may instead send the requested texture data to the storage controller. Once data is received at GPU accelerator 703 or storage controller, data may be sent 710 to computer system 701.

[0036] Figure 8 This is a timing diagram of the operation of an improved storage device with an NPU according to various aspects of this disclosure. In this embodiment, the NPU generates a machine learning model trained with a machine learning algorithm to generate derived assets. Computer system 801 is communicatively coupled to improved storage device 820. Computer system 801 can send a request for an asset to NPU 802. Computer system 801 may be configured with sufficient programming to support receiving asset derivatives of the requested asset from the NPU. In some embodiments, the computer system can request assets from the NPU when consistency in asset appearance is not required. For example, but not limited to, in the context of video games, the computer system can request background assets from the NPU that the user may not frequently examine.

[0037] NPU 802 can receive a request for asset 804 and initiate the required NN model to generate a derived asset from the requested asset. Alternatively, NPU 802 can be pre-configured to generate derived assets from the base asset. In either case, NPU 802 sends a request 805 for the base asset to flash memory 803. Flash memory 803 then sends the base asset 806 to NPU 802. Upon receiving the base asset, NPU 802 generates the derived asset using a trained machine learning model. Once the derived asset is generated, NPU 802 sends the derived asset 807 to computer system 801.

[0038] In other alternative implementations, the underlying asset may be (or may include) a text description of an image, and the neural network may generate derivatives in the form of one or more images derived from the text description. In other alternative implementations, the underlying asset may be (or may include) a text description of a digital object, and the neural processor may generate derived assets from the text description in the form of a three-dimensional digital representation of the object. In other alternative implementations, the underlying asset may be (or may include) an image of a digital object, and the neural processor may generate derived assets from the image of the digital object in the form of a three-dimensional representation of the digital object image. In other specific implementations, the flash memory may contain a script file including instructions for generating derived assets, and the neural processor may use the script file to generate derived assets from the underlying asset.

[0039] According to various aspects of this disclosure, an NPU can implement machine learning models to generate derived assets from an underlying asset stored in memory. An example of a machine learning model that can be trained to generate derived assets from an underlying asset is a diffusion model. A diffusion model trains a neural network to predict images based on a noise distribution. Initially, the diffusion model is trained to remove noise added to a clean image. Once fully trained, the model is responsible for generating images from random noise. For more information on diffusion models, see: Yang, Ling, “Diffusion Models: A Comprehensive Survey of Methods and Applications”, ArXiv, 2209.00796, published September 2, 2022, available at https: / / arxiv.org / abs / 2209.00796, the contents of which are incorporated herein by reference. Another generative model is an autoencoder. An autoencoder is a neural network layout that has an encoder network involved in dimensionality reduction of the output embedding and a decoder network that uses the embedding to predict a synthetic output. The autoencoder neural network outputs feature-length embeddings, and the decoder includes a neural network that uses those feature-length image embeddings to generate one or more synthetic assets. For more information on autoencoder asset generation, see: Huiwen, Chang, “Muse: Text-To-Image Generation via Masked Generative Transformers”, ArXiv, arXiv:2301.00704, published January 2, 2023, available at https: / / arxiv.org / abs / 2301.00704, the contents of which are incorporated herein by reference. Another generative approach is Neural Radiance Fields (NeRF), which generates 3D representations of multiple image views. More information on NeRF can be found at: Mildenhall, Ben, “NeRF: Representation Scenes as Neural Radiance Fields for View Synthesis”, ArXiv, arXiv:2003.08934, published March 19, 2020, available at https: / / arxiv.org / abs / 2003.08934, the contents of which are incorporated herein by reference.

[0040] As the preceding discussion has shown, storage devices can improve the functionality of connected computer systems by adding hardware accelerators. These accelerators selectively index and retrieve data stored in flash memory, reducing the amount of data the computer system needs to read from and / or retrieve from flash memory. Furthermore, improved storage devices with NPUs can leverage generative machine learning models implemented by the NPU to generate multiple different derived assets. This reduces the amount of data the computer system needs to store and / or retrieve from flash memory to generate different assets, as a single underlying asset can be used to generate multiple different assets.

[0041] While the foregoing is a complete description of preferred embodiments of the invention, various alternatives, modifications, and equivalents may be used. Therefore, the scope of the invention should not be determined by reference to the foregoing description, but rather by reference to the full scope of the appended claims and their equivalents. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the following claims, the indefinite article “a” or “an” refers to the number of one or more items following the article, unless otherwise expressly stated. The appended claims should not be construed as including a component plus a functional limitation, unless such limitation is expressly stated in a given claim using the phrase “component for…”.

Claims

1. A storage device, comprising: Storage controller; Flash memory, communicatively coupled to the storage controller; A hardware accelerator communicatively coupled to the flash memory and the storage controller, wherein the hardware accelerator is configured to selectively retrieve data in response to a request for data stored in the flash memory.

2. The storage device according to claim 1, wherein, The storage controller, the flash memory, and the hardware accelerator are located on the same circuit board.

3. The storage device of claim 1 further includes a central processing unit and a memory located on the same circuit board as the storage controller, flash memory and the hardware accelerator.

4. The storage device of claim 1 further includes a physical interface communicatively coupled to the storage controller and the hardware accelerator.

5. The storage device according to claim 4, wherein, The physical interface is an M.2 connector.

6. The storage device according to claim 4, wherein, The physical interface is a Universal Serial Bus connector.

7. The storage device according to claim 4, wherein, The request for the data is received from a computer system that is communicatively coupled to the storage device through the physical interface.

8. The storage device according to claim 4, wherein, The request for the data is received by the storage controller, and the storage controller transmits the request to the hardware accelerator.

9. The storage device according to claim 1, wherein, The hardware accelerator is a graphics processing unit accelerator, and the data is a texture, wherein the graphics processing unit accelerator is configured to retrieve the data using an index, wherein the index corresponds to a level of detail, and the texture is stored in the flash memory indexed by level of detail.

10. The storage device according to claim 9, wherein, The graphics processing unit accelerator also includes at least one of a hardware video decoder module and a hardware video encoder module.

11. The storage device according to claim 9, wherein, The graphics processing unit accelerator also includes a hardware graphics processing pipeline stage.

12. The storage device according to claim 1, wherein, The hardware accelerator is a field-programmable gate array (FPGA).

13. The storage device according to claim 1, wherein, The storage controller is an NVME controller or a USB controller.

14. The storage device according to claim 1, wherein, The hardware accelerator is also configured to store the data in the flash memory using an index in response to a storage request for the data.

15. The storage device according to claim 14, wherein, The hardware accelerator is also configured to generate index entries in the flash memory for the data stored therein.

16. The storage device according to claim 1, wherein, The hardware accelerator uses an index based on the file type of the data to selectively retrieve data stored in the flash memory.

17. The storage device according to claim 1, wherein, The hardware accelerator is also configured to compress the data.

18. The storage device according to claim 1, wherein, The data is in a vector-oriented data format, and the hardware accelerator is also configured to perform a search using the data in the vector-oriented data format.

19. The storage device of claim 1, wherein the hardware accelerator is communicatively coupled to the second hardware accelerator via at least a first computer system over a network.

20. The storage device according to claim 19, wherein, The hardware accelerator is also configured to send the data to the second hardware accelerator via the network.

Images