Systems, methods, and apparatus for memory loading in storage device

Abstract

A device may include a memory medium; a storage medium; a buffer; and at least one circuit configured to perform one or more operations, the one or more operations including: receiving memory address information; storing the memory address information in a buffer; determining that the data can be loaded to the memory medium; and loading data corresponding to the memory address information in the buffer from the storage medium to the memory medium. In some aspects, the memory address information is first memory address information; the data is first data; and the at least one circuit is further configured to perform one or more operations, the one or more operations comprising: receiving a memory access request comprising second memory address information; determining to load second data based on the memory access request; and loading the second data from the storage medium to the memory medium based on the second memory address information.

Description

[0001] Citation of Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application Serial No. 63 / 544,438, filed on October 16, 2023, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to storage devices, and more particularly to systems, methods, and apparatus for memory loading in storage devices. Background Art

[0004] The storage device may include a memory media (e.g., a cache media) and a storage media. When a host sends a request for data to a storage device, the storage device may first check the memory media for data. However, if the data is not found on the memory media, a cache miss may occur, and the storage device may search for the data on the storage media.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the principles of the present invention and therefore it may contain information that does not constitute the prior art. Summary of the invention

[0006] In some aspects, the technology described herein relates to a device, the device comprising: a memory medium; a storage medium; a buffer; and at least one circuit configured to perform one or more operations, the one or more operations comprising: receiving memory address information; storing the memory address information in the buffer; determining that data can be loaded into the memory medium; and loading data corresponding to the memory address information in the buffer from the storage medium to the memory medium. In some aspects, the memory address information is first memory address information; the data is first data; and the at least one circuit is further configured to perform one or more operations, the one or more operations comprising: receiving a memory access request including second memory address information; determining to load second data based on the memory access request; and loading the second data from the storage medium to the memory medium based on the second memory address information. In some aspects, loading the data comprises: obtaining at least one address from the memory address information; and adding at least one address to the buffer. In some aspects, the memory address information comprises an indication of the length of the address and at least one address. In some aspects, the memory address information comprises at least one address and an indication of the number of addresses. In some aspects, the at least one circuit is further configured to perform one or more operations, the one or more operations comprising: receiving an instruction to make the buffer available; and making the buffer available. In some aspects, the at least one circuit is further configured to perform one or more operations, the one or more operations including: receiving an instruction to make a memory medium available; and making the memory medium available. In some aspects, the at least one circuit includes a control register, the control register is configured to perform one or more operations, the one or more operations including: determining a state of a buffer based on the control register; and modifying an operation of the at least one circuit based on the state.

[0007] In some aspects, the technology described herein relates to a method, the method comprising: receiving memory address information; storing the memory address information in a buffer; determining that data can be loaded into a memory medium; and loading the data from the storage medium to the memory medium based on the memory address information in the buffer. In some aspects, the memory address information is first memory address information; the data is first data; and the method further comprises: receiving a memory access request including second memory address information; determining to load second data based on the memory access request; and loading the second data from the storage medium to the memory medium based on the second memory address information. In some aspects, loading the data comprises: obtaining at least one address from the memory address information; and adding the at least one address to the buffer. In some aspects, the memory address information comprises an indication of the length of the address and at least one address. In some aspects, the memory address information comprises at least one address and an indication of the number of addresses. In some aspects, the method further comprises: receiving an instruction to make the buffer available; and making the buffer available. In some aspects, the method further comprises determining a state of the buffer; and modifying operations on the buffer based on the state.

[0008] In some aspects, the technology described herein relates to a system comprising: a host device, the host device comprising an application and a thread, wherein the thread comprises memory address information of the application; and a storage device, comprising a memory medium; a storage medium; a buffer; and at least one circuit configured to perform one or more operations, the one or more operations comprising: receiving memory address information; storing the memory address information in the buffer; determining that data can be loaded into the memory medium; and loading the data from the storage medium to the memory medium based on the memory address information in the buffer. In some aspects, the at least one circuit is further configured to perform one or more operations, the one or more operations comprising: receiving a memory access request comprising second memory address information; determining to load second data based on the memory access request; and loading the second data from the storage medium to the memory medium based on the second memory address information. In some aspects, the memory address information comprises an indication of the length of the address and at least one address. In some aspects, the memory address information comprises at least one address and an indication of the number of addresses. In some aspects, the at least one circuit is further configured to perform one or more operations, the one or more operations comprising: receiving an instruction to make the buffer available; and making the buffer available. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The drawings are not necessarily drawn to scale, and throughout the drawings, for illustrative purposes, elements of similar structure or function may generally be represented by the same reference numerals or portions thereof. The drawings are intended only to facilitate the description of the various embodiments described herein. The drawings do not describe every aspect of the teachings disclosed herein and do not limit the scope of the claims. In order to prevent the drawings from becoming obscure, not all components, connections, etc. may be shown, and not all components may have reference numerals. However, the mode of component configuration can be easily apparent from the drawings. The drawings, together with the specification, illustrate example embodiments of the present disclosure and, together with the specification, are used to explain the principles of the present disclosure.

[0010] Figure 1 An embodiment of a storage device scheme according to an example embodiment of the present disclosure is shown.

[0011] Figure 2 An embodiment of a storage device according to an example embodiment of the present disclosure is shown.

[0012] Figure 3 An example of a pre-fetch process according to an example embodiment of the present disclosure is shown.

[0013] Figure 4 An embodiment of another storage device according to an example embodiment of the present disclosure is shown.

[0014] Figure 5 A method for loading data to a buffer of a storage device using a prefetcher according to an example embodiment of the present disclosure is shown.

[0015] Figure 6 An example operating environment for loading data into a buffer of a storage device using a prefetcher according to an example embodiment of the present disclosure is shown.

[0016] Figure 7a An example of determining an order of processing requests according to an exemplary embodiment of the present disclosure is shown.

[0017] Figure 7b Another example of determining an order of processing requests according to an example embodiment of the present disclosure is shown.

[0018] Figure 8 A flowchart of pre-fetching data according to an example embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0019] A memory medium (e.g., a cache medium) can be used to improve the overall performance of a storage device. For example, a storage device can fill data in a memory medium to respond to future requests for the data faster than if the data were only located in the storage medium. If a host requests data in a memory medium (e.g., a cache hit), the data can be retrieved from the memory medium. If the data is not found in the memory medium (e.g., a cache miss), the data can be retrieved from the storage medium. In some embodiments, as more cache hits occur, performance can be improved because data can be retrieved from the memory medium. Typically, a memory medium can include a memory (e.g., a dynamic random access memory (DRAM)) that allows faster reading and writing than a storage medium such as a non-AND (NAND) flash memory device. A storage device according to an example embodiment of the present disclosure may include a memory medium (e.g., a DRAM) and a storage medium (e.g., a NAND flash memory device).

[0020] In some embodiments, the performance of a storage device may benefit from parallelism (e.g., processing multiple commands simultaneously). For example, when a NAND flash memory device is being accessed (e.g., a load / store request), other commands may be processed simultaneously, thereby alleviating some of the delays associated with NAND-based media. In some embodiments, it may be beneficial for an input / output (I / O) queue of a storage device to remain full and / or close to capacity. For example, by keeping an I / O queue full, commands may be received for processing by the storage device so that the storage device may not wait to receive additional commands. However, the ability to process multiple commands from an I / O queue may not be fully utilized because the queue may not have the capacity to handle many load / store requests.

[0021] In addition, when the memory medium is not effectively filled, the access pattern to the storage device may be random, resulting in a lower cache hit rate. When a cache miss occurs, data may be loaded from the storage medium to the memory medium, which may result in a longer delay than retrieving data from the memory medium. In some embodiments, a way to mitigate the delay may be to have multiple processes on the application side that can read data simultaneously. However, the number of processes may be limited by computing resources (e.g., central processing unit (CPU) cores), resulting in insufficient I / O of the storage device. In some embodiments, dedicating more processes to data loading may result in insufficient computing processes, which in turn may reduce overall performance (e.g., fewer computing processes may reduce the number of queries per second ("QPS")). In addition, scaling the number of processes may not improve performance to offset the cost of adding additional computing resources to the device. Therefore, in some embodiments, in order to increase the cache hit rate, an in-device hardware prefetcher may be introduced to the storage device.

[0022] In some embodiments, in-device hardware prefetching can improve cache hit rates, resulting in increased input / output operations per second (IOPS), while also reducing latency of the storage device. Thus, in some embodiments, multiple threads for loading data may not be added, which may result in reduced CPU resource consumption compared to devices that do not perform in-device hardware prefetching.

[0023] According to an embodiment of the present disclosure, the storage device may include a prefetcher. In some embodiments, the storage device may retrieve address information from a host for use in an application. In some embodiments, the storage device may store the address information in the prefetcher, and retrieve the address information from the prefetcher to fill a cache on the storage device.

[0024] The present disclosure covers many aspects related to devices with memory and storage configurations. The aspects disclosed herein can have independent utility and can be embodied individually, and not every embodiment can utilize every aspect. In addition, these aspects can also be embodied in various combinations, some of which can amplify some benefits of various aspects in a collaborative manner.

[0025] For purposes of illustration, some embodiments may be described in the context of some specific implementation details, such as a device implemented as a storage device that may use specific interfaces, protocols, etc. However, aspects of the disclosure are not limited to these or any other implementation details.

[0026] Figure 1 An embodiment of a storage device solution according to an exemplary embodiment of the present disclosure is shown. Figure 1 The embodiment shown in may include one or more host devices 100 and one or more storage devices 150 configured to communicate using one or more communication connections 110 .

[0027] In some embodiments, the host device 100 may be implemented with any component or combination of components that may utilize one or more features of the storage device 150. For example, the host may be implemented with one or more of a server, a storage node, a computing node, a central processing unit (CPU), a workstation, a personal computer, a tablet computer, a smart phone, etc., or a plurality and / or combination thereof.

[0028] In some embodiments, storage device 150 may include communication interface 130, memory 180 (some or all of which may be referred to as device memory), one or more computing resources 170 (which may also be referred to as computing resources), device controller 160, and / or device functional circuitry 190. In some embodiments, device controller 160 may control the overall operation of storage device 150, including any operations, features, etc. described herein. For example, in some embodiments, device controller 160 may parse, process, invoke, etc., commands received from host device 100.

[0029] In some embodiments, the device functional circuit 190 may include any hardware for implementing the primary functions of the storage device 150. For example, the device functional circuit 190 may include storage media, such as magnetic media (e.g., if the storage device 150 is implemented as a hard disk drive (HDD) or a tape drive), solid-state media (e.g., one or more flash memory devices), optical media, etc. For example, in some embodiments, the storage device may be implemented at least in part as a solid-state drive (SSD) based on NAND flash memory, a persistent memory (PMEM) such as a cross-grid non-volatile memory, a memory with a body resistance change, a phase change memory (PCM), or any combination thereof. In some embodiments, the device controller 160 may include a media translation layer, such as a flash translation layer (FTL), for interfacing with one or more flash memory devices. In some embodiments, the storage device 150 may be implemented as a computational storage drive, a computational storage processor (CSP), and / or a computational storage array (CSA).

[0030] As another example, if the storage device 150 is implemented as an accelerator, the device functional circuitry 190 may include one or more accelerator circuits, memory circuits, and the like.

[0031] The computing resources 170 may be implemented with any component or combination of components that can perform operations on data that can be received, stored, and / or generated at the storage device 150. Examples of computing engines may include combinatorial logic, sequential logic, timers, counters, registers, state machines, complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), embedded processors, microcontrollers, central processing units (CPUs) such as complex instruction set computers (CISC) processors (e.g., x86 processors) and / or reduced instruction set computers (RISC) processors such as ARM processors), graphics processing units (GPUs), data processing units (DPUs), neural processing units (NPUs), tensor processing units (TPUs), etc., which may run instructions stored in any type of memory and / or implement any type of operating environment such as containers, virtual machines, operating systems such as Linux, extended Berkeley packet filters (eBPF) environments, etc., or a combination thereof.

[0032] In some embodiments, memory 180 may be used, for example, by one or more of computing resources 170 to store input data, output data (e.g., calculation results), intermediate data, transitional data, etc. Memory 180 may be implemented, for example, with volatile memory such as DRAM, static random access memory (SRAM), etc., as well as any other type of memory such as non-volatile memory.

[0033] In some embodiments, the memory 180 and / or computing resources 170 may include software, instructions, programs, code, etc. that may be executed and / or run using one or more computing resources (e.g., hardware (HW) resources). Examples may include software implemented in any language such as assembly language, C, C++, etc., binary code, FPGA code, one or more operating systems, kernels, environments such as eBPF, etc. The software, instructions, programs, code, etc. may be stored in, for example, a repository in the memory 180 and / or computing resources 170. In some embodiments, the software, instructions, programs, code, etc. may be downloaded, uploaded, side-loaded, pre-installed, built-in, etc. to the memory 180 and / or computing resources 170. In some embodiments, the storage device 150 may receive one or more instructions, commands, etc. to select, enable, activate, run, etc. the software, instructions, programs, code, etc. Examples of computing operations, functions, etc. that can be implemented by memory 180, computing resources 170, software, instructions, programs, codes, etc. may include any type of algorithm for artificial intelligence (AI), machine learning (ML), neural networks, etc., data movement, data management, data selection, filtering, encryption and / or decryption, compression and / or decompression, checksum calculation, hash value calculation, cyclic redundancy check (CRC), weight calculation, activation function calculation, training, inference, classification, regression, etc.

[0034] In some embodiments, the communication interface 120 at the host device 100, the communication interface 130 at the storage device 150, and / or the communication connection 110 may be implemented using any type of interface, protocol, etc. and / or implemented with one or more interconnects, one or more networks, a network of networks (e.g., the Internet), etc., or a combination thereof. For example, the communication connection 110 and / or one or more of the communication interfaces 120 and / or 130 may be implemented and / or implemented with any type of wired and / or wireless communication medium, interface, network, interconnect, protocol, etc., including Peripheral Component Interconnect Express (PCIe), NVMe, NVMe over Fabric (NVMe-oF), Compute Express Link (CXL) and / or coherent protocols (such as CXL.mem, CXL.cache, CXL.io, etc.), Gen-Z, Open Coherent Accelerator Processor Interface (OpenCAPI), Cache Coherent Interconnect for Accelerators (Cache Coherent Interconnect for The communication connection 110 may include any generation of wireless networks including 2G, 3G, 4G, 5G, 6G, etc., Wi-Fi, Bluetooth, near field communication (NFC), etc., or any combination thereof. In some embodiments, the communication connection 110 may include one or more switches, hubs, nodes, routers, etc.

[0035] In some embodiments, the storage device 150 can be implemented in any physical form factor. Examples of form factors may include 3.5 inches, 2.5 inches, 1.8 inches, etc., storage device (e.g., storage drive) form factors, M.2 device form factors, Enterprise and Data Center Standard Form Factor (EDSFF) (which may include, for example, E1.S, E1.L, E3.S, E3.L, E3.S2T, E3.L 2T, etc.), Add-in Card (AIC) (e.g., PCIe card (e.g., PCIe expansion card) form factors, including half-height (HH), half-length (HL), half-height, half-length (HHHL), etc.), Next Generation Small Form Factor (NGSFF), NF1 form factor, Compact Flash (CF) form factor, Secure Digital (SD) card form factor, Personal Computer Memory Card International Association (PCMCIA) device form factor, etc., or combinations thereof. Any computing device disclosed herein can be connected to a system using one or more connectors, such as a SATA connector, a SCSI connector, a SAS connector, an M.2 connector, an EDSFF connector (e.g., 1C, 2C, 4C, 4C+, etc.), a U.2 connector (which may also be referred to as an SSD Form Factor (SSF) SFF-8639 connector), a U.3 connector, a PCIe connector (e.g., a card edge connector), and the like.

[0036] Any storage device disclosed herein may be used in conjunction with one or more personal computers, smart phones, tablet computers, servers, server chassis, server racks, modems, data centers, edge data centers, mobile edge data centers, and / or any combination thereof.

[0037] In some embodiments, storage device 150 may be implemented with any device that may include or access memory, storage media, etc. to store data that may be processed by one or more computing resources 170. Examples may include memory extension and / or buffer devices, such as CXL Type 2 and / or CXL Type 3 devices, as well as CXL Type 1 devices that may include memory, storage media, etc.

[0038] Figure 2 An embodiment of a storage device according to an example embodiment of the present disclosure is shown. Figure 2A host device 200 and a storage device 250 are shown. The storage device 250 may include a memory medium 260 and a storage medium 270. In some embodiments, the host device 200 may run an application that can generate multiple read threads (e.g., read thread 1, read thread 2, ... read thread N) for data. In some embodiments, the host device 200 may communicate with the storage device 250 using a memory access protocol (e.g., CXL.mem protocol). In some embodiments, the host device 200 may request data from the storage device 250. In some embodiments, the host device may not know where the data is located on the storage device, and if the data is located in the memory medium (e.g., cache hit), the storage device may return the data from the memory medium 260, or if the data is not found on the memory medium 260, then return the data from the storage medium 270. In some embodiments, the memory medium 260 may be a DRAM. In some embodiments, the storage medium may be a NAND. In some embodiments, retrieving data from the memory medium 260 may be faster than retrieving data from the storage medium, and therefore, it may be necessary to pre-fetch data from the storage medium 270 to the memory medium 260 to increase the number of cache hits.

[0039] Figure 3 An example of a pre-fetch process according to an exemplary embodiment of the present disclosure is shown. Figure 3 The embodiments shown in can be used to implement and / or can be used, for example, in Figure 2 , where similar elements may be indicated by reference numerals ending with and / or containing the same numbers, letters, etc.

[0040] In some embodiments, host device 200 may request storage device 250 to prefetch data from storage medium 270 to memory medium 260 using a memory access protocol (e.g., CXL.mem). For example, the CXL.mem protocol may allow the host to send an address to be fetched from memory medium 260. However, due to the delay in loading data from storage medium 270 to memory medium 260, the data may be accessed by the host device before the prefetch can be completed. Figure 1, host device 200 (shown in the upper left box) shows a current position at block 0 and pre-fetch requests for blocks 3, 4, and 6. In other words, host device 200 can access memory at block 0 while also requesting that memory medium 260 be populated with blocks 3, 4, and 6. However, in some embodiments, due to the latency associated with moving data from the storage medium, when the host attempts to access block 3 (shown in the lower left box), it may not yet be populated in memory medium 260. Therefore, the storage device may process the request from storage medium 270 instead of memory medium 260, resulting in increased latency due to the slower speed of storage medium 270.

[0041] Figure 4 An embodiment of a storage device according to an example embodiment of the present disclosure is shown. Figure 4 The embodiments shown in can be used to implement and / or can be used, for example Figure 2 , where similar elements may be indicated by reference numerals ending with and / or containing the same numbers, letters, etc. In some embodiments, storage device 250 may be configured to use and / or be compatible with various aspects of a memory access protocol such as CXL. In some embodiments, memory medium 260 may be configured as a cache and storage medium 270 may be configured to store data. For example, storage medium 270 may store persistent data, and at least a portion of the data may be written to memory medium 260.

[0042] In this example, the memory medium 260 may be a DRAM device cache and the storage medium 270 may be a NAND-based medium (e.g., an NVMe storage device). However, in other embodiments, the memory medium 260 may be implemented with other types of memory media (e.g., volatile media that may have relatively high access speeds and low latency). In other embodiments, the storage medium 270 may be implemented with other types of storage media (e.g., non-volatile media that may have relatively high capacity and / or relatively low cost).

[0043] In some embodiments, the host device 200 may include an application 410 and a prefetch thread 420. In some embodiments, the prefetch thread 420 may be a part of the application 410, while in other embodiments, the prefetch thread 420 may be a separate component of the host device 200. In some embodiments, the application 410 may generate multiple read threads (e.g., read thread 1, read thread 2, ..., read thread N) for data. In some embodiments, the host device 200 may pass a read request corresponding to the read thread to the storage device 250 using a memory access protocol (e.g., CXL.mem). In some embodiments, the prefetch thread 420 may determine what data the application 410 will use, and pass information of the data (e.g., memory address information) to the prefetcher 452. In some embodiments, the host device 200 may pass the memory address information to the prefetcher 452 using the prefetch library 422 and the storage device driver 424 using a memory access protocol (e.g., CXL.IO). In some embodiments, the memory access protocol for the read request and the memory access protocol for the prefetcher may be different protocols.

[0044] In some embodiments, the storage device 250 may also include at least one circuit (e.g., a cache controller 456 and a storage medium controller 458). In some embodiments, the cache controller 456 may be used to manage the storage medium 260. For example, the cache controller 456 may be used to copy data to the storage medium 260, for example, to retrieve data from the storage medium 270 to fill the storage medium 260. In some embodiments, the cache controller 456 may communicate with the storage medium controller 458 to retrieve data from the storage medium 270. In some embodiments, the storage medium controller 458 may be used to handle read and write operations to the storage medium 270. For example, if the data is not found in the storage medium 260 (e.g., a cache miss), the cache controller 456 may issue a read request (e.g., a NAND read request) to the storage medium controller 458, and the storage medium controller 458 may read the data from the storage medium 270. In some embodiments, the read request may be an I / O request using a memory access protocol. In some embodiments, cache controller 456 can receive data from storage medium 270 and write it to storage medium 260 (e.g., DRAM cache write request). Additional examples of operations of cache controller 456 can include implementing one or more cache maps, data write policies (e.g., write-through, write-back, etc.), cache replacement policies, etc. In some embodiments, storage medium controller 458 can be a NAND controller (e.g., a NAND channel controller) that can control a NAND flash memory that can be used to implement storage medium 270.

[0045] In some embodiments, the storage device 250 may include at least one circuit (e.g., a prefetcher 452 and an arbiter 454). In some embodiments, the prefetcher 452 may receive memory address information from a host. For example, the memory address information may be an address of data for use by an application or a range of addresses of data. In some embodiments, the memory address information may be used to copy data from the storage medium 270 to the memory medium 260. In some embodiments, the prefetcher 452 may store the address information in a buffer (e.g., a prefetch queue) and retrieve (e.g., dequeue) the memory address information from the buffer to request data from the memory medium 260. In some embodiments, if the data is not found in the memory medium 260, then the data may be copied from the storage medium 270 to the memory medium 260. In some embodiments, the arbiter 454 may decide the order in which to process the requests. For example, if an I / O request is received from the host, the arbiter 454 may decide whether to process the I / O request or process the request (e.g., address) from the prefetcher 452. Although, in Figure 4 In the embodiment, the pre-fetcher 452 is included in the storage device 250, but the pre-fetcher can also be part of the host or the second device, wherein the host or the second device can send the address information for filling the storage medium 260. In addition, the pre-fetcher 452 can be implemented as hardware, software, or at least partially as software and hardware. In addition, although the pre-fetcher 452 is from Figure 4 The host in receives address information, but the address information can come from any other source that can consider the address information of the host. For example, the address information can be determined by the usage pattern of the application, from historical data, or can be determined using AI. Any source of address information can be considered within the scope of the present disclosure.

[0046] In some embodiments, in order to obtain an address from the memory address information, the prefetcher 452 can parse the address from the memory address information. For example, the memory address information may include an indication of the length of the address (e.g., the number of addresses) and at least one address. In some embodiments, if the memory address information includes at least one address, then the at least one address can be parsed by the prefetcher 452 to the prefetch queue. In some embodiments, the memory address information may alternatively include an indication of at least one address and a range of addresses. For example, some applications can process data in batches. Therefore, by prefetching a range of addresses, data can be loaded into the cache. In some embodiments, if the memory address information is a range of addresses, the prefetcher 452 can determine the starting address from the memory address information, and the addresses starting from the starting address can be added to the prefetch queue.

[0047] In some embodiments, prefetcher 452 may be any hardware implementation suitable for receiving addresses and passing addresses to cache controller 456. For example, prefetcher 452 may be a field programmable gate array (FPGA) implementation. Prefetcher 452 may also be implemented on an application specific integrated circuit (ASIC), a system on a chip (SoC), or any other suitable hardware architecture.

[0048] In some embodiments, a load balancing scheme may be used by the arbitrator 454 to determine the order in which requests are processed. In some embodiments, the load balancing scheme may allow the storage device to fairly process requests from the host and the prefetcher so that requests from both the host and the prefetcher can be efficiently processed. Any load balancing scheme may be used by the arbitrator 454 to determine the order in which requests are processed. For example, the arbitrator 454 may use weighted round-robin for the load balancing scheme. In some embodiments, requests from the host and requests from the prefetcher may be given weights, and the arbitrator 454 may use the weights to determine which requests are processed in order. It is within the scope of the present disclosure to use any appropriate load balancing scheme for the arbitrator 454.

[0049] In some embodiments, for a CXL-compatible storage device, an I / O request from a host may include a CXL.mem request, and a prefetch request from a host may be a CXL.io request. Typically, an I / O request (e.g., load / store) allows a host to access a storage device. However, a host may not be able to directly access a cache using an I / O request. Instead, a host may make a request for data, which may be processed by a cache and / or a storage medium. However, by using a protocol such as CXL.io, a host may directly access a cache (e.g., access data using address information). In some embodiments, when an address is not found on a cache (e.g., a cache miss occurs), data may be copied from storage medium 270 to memory medium 260.

[0050] In this example, the storage device 250 may implement one or more memory access protocols, but in other embodiments may implement or may be implemented with any other interface, protocol, etc. that may enable a user to access data as memory (e.g., DDR, OMI, Gen-Z, DMA, RDMA, etc.). The CXL (or other) protocol may operate with any underlying transport scheme (e.g., physical layer, transport layer, etc.), including, for example, PCIe, Ethernet, InfiniBand, etc.

[0051] In some embodiments, prefetcher 452 may have its own prefetch queue for storing addresses for prefetching. In some embodiments, the prefetch queue may be a circular data buffer.

[0052] In some embodiments, the prefetcher 452 may support additional commands. For example, a buffer enable or clear command (e.g., a command to clear the buffer), a cache enable or invalid command (e.g., to invalidate or clear data in the cache), and a query command (e.g., to query the state of the circular data buffer) may also be provided. In some embodiments, if the storage device 250 receives a buffer enable or clear command, the prefetcher 452 may enable (e.g., clear) the prefetch queue.

[0053] In some embodiments, the cache controller 456 may include instructions to make data in the memory medium 260 available (e.g., cleared) (e.g., an invalidate command) and to make the memory medium 260 available (e.g., clear the data in the memory medium). For example, if the data in the memory medium 260 is no longer in use, the memory medium 260 may be made available (e.g., cleared), thereby freeing up memory in the memory medium 260 for writing. Thus, the cache controller 456 may write data to the memory medium 260 when the memory medium 260 is available (e.g., the data is cleared).

[0054] In some embodiments, storage device 250 may include a prefetcher control register. In some embodiments, the prefetcher control register may be used to determine whether the queue is full. In some embodiments, if the queue is full, the prefetcher 452 may wait for additional memory address information to be loaded into the prefetch queue. This prevents the existing memory address information in the prefetch queue from being overwritten and / or causing an error in the system to attempt to write the memory address information into the prefetch queue. In some embodiments, if the queue is not full, the memory address information may be written into the prefetch queue. In some embodiments, whenever the memory address information is written into the prefetch queue or sent to the cache controller 456, the prefetch control register may be updated.

[0055] Figure 5 An example of a method for loading data to a buffer of a storage device using a prefetcher according to an example embodiment of the present disclosure is shown.

[0056] At 510, the prefetcher may receive memory address information. For example, the host may send a request to the prefetcher to prefetch data. In some embodiments, the memory address information may include an indication of the length of the address (e.g., the number of addresses in the request), and one or more memory addresses. In some embodiments, the address information may alternatively include a first address and an indication of the number of addresses (e.g., a range of addresses).

[0057] At 520, the prefetcher may store the memory address information in a buffer. For example, if the memory address information includes one or more addresses, the addresses may be parsed by the prefetcher into the buffer. In some embodiments, if the memory address information is a range of addresses, the prefetcher may determine the starting address (and subsequent addresses) from the memory address information. In some embodiments, the prefetcher may add the one or more addresses to the buffer.

[0058] At 530, data may be loaded into the memory medium based on the address information in the buffer. For example, the prefetcher may retrieve the address from the buffer. In some embodiments, the address in the prefetch queue may be used to find / load data into the memory medium, as described above with respect to Figure 3 As described. In some embodiments, the arbitrator can receive an address from the queue and determine whether to process an address from the pre-fetch queue or another I / O request (e.g., from the host). In some embodiments, if the arbitrator selects an address from the pre-fetch queue, the storage device can look for the data in the memory medium. If it cannot find the data in the memory medium, the data can be copied from the storage medium to the memory medium. In some embodiments, the host can know that the above steps are being performed, for example, an indicator may not be returned to the host to indicate that the data has been loaded into the memory medium. Therefore, the pre-fetch operation can be transparent to the host, and the host can receive data from the memory medium.

[0059] In some embodiments, if the data is no longer needed or additional data is needed, the prefetch queue may be made available (eg, cleared), thereby allowing new memory address information to be loaded into the prefetch queue.

[0060] Figure 6 An example operating environment for loading data into a buffer of a storage device using a prefetcher according to an example embodiment of the present disclosure is shown. Figure 6 The embodiments shown in can be used to implement and / or can be used, for example Figure 2 , where similar elements may be indicated by reference numerals ending with and / or containing the same numbers, letters, etc.

[0061] In some embodiments, the host 610 (e.g., using a background prefetch thread) may send a prefetch request to the storage device 600. For example, the host 610 may send a number of addresses (e.g., an indication of the length of the address in the request) and one or more addresses to the storage device 600. In some embodiments, the host 610 may send a first address and an indication of the number of addresses (e.g., a range of addresses) to the storage device 600. In some embodiments, the host 610 may send a prefetch request including the address of the data to be prefetched to the device using a cache coherent protocol. In some embodiments, the host 610 may know what address to prefetch from the storage device 600. In particular, for some applications, data may be stored to the storage device 600 in such a way that the application and / or the host may also know how to load the data. In other words, the address information may be predicted by the application and / or the host. For example, the prefetcher may be used with applications having multiple threads for load / store operations and applications that know data access patterns, for example, for machine learning (ML) applications such as deep learning recommendation models (DLRMs) that process batch data, or bioinformatics. In some embodiments, an AI (eg, an ML model) may be used to determine the address of a prefetch thread.

[0062] In some embodiments, an application programming interface (API) may be provided to prefetch data. For example, the host may use a cache coherence protocol command to prefetch data. In some embodiments, the command to prefetch data may not be limited to any particular size. However, for the purpose of illustrating some aspects of the present disclosure, in an example embodiment, the request may be 4 kilobytes (KB). In some embodiments, on the other hand, a load / store request may be of a different size, for example, 64 bytes. Therefore, in contrast to a load / store request, a prefetch request may include more addresses to be prefetched.

[0063] In some embodiments, the storage device 600 may include a prefetcher 650, and the prefetcher 650 may include a buffer 652 (e.g., a circular data buffer). In some embodiments, addresses may be parsed from a prefetch request and added to the buffer 652. In some embodiments, as new addresses are read in, the addresses may be added to the end of the buffer 652. In some embodiments, the storage device 600 may include a control register 660. In some embodiments, the control register 660 may include a head pointer 662 indicating the start position of the buffer 652 (e.g., the first address), a tail pointer 664 indicating the end position of the buffer 652 (e.g., the last address), and a length of a register 666 indicating the number of entries in the register. In some embodiments, the control options (e.g., clearing) of the prefetcher may be implemented by a memory mapped I / O ("MMIO"). Thus, the host may send an I / O request to clear the buffer 652, which, when received, may clear the addresses in the buffer 652. In some embodiments, the next few batches of data may be prefetched while the current batch is being processed. In some embodiments, buffer 652 may be larger (eg, 32k entries) to hold addresses to be pre-fetched. In some embodiments, control register 660 may be implemented in an FPGA, and buffer 652 may be implemented in SRAM or DRAM.

[0064] In some embodiments, other commands may also be supported. For example, a prefetch status request may allow the host to query the status of the prefetcher. In some embodiments, if the prefetch status request indicates that the buffer 652 is full, the host may wait to send a prefetch request.

[0065] Figure 7a and Figure 7b An example of determining the order of processing requests according to an exemplary embodiment of the present disclosure is shown. Figure 7a and Figure 7b The embodiments shown in can be used to implement and / or can be used, for example, in Figure 4, wherein similar elements may be indicated by reference numerals ending with and / or containing the same numbers, letters, etc. In some embodiments, request 710 may be an I / O request from a host. In some embodiments, request 750, request 760, and request 770 may be addresses on a pre-fetch queue. In some embodiments, arbitrator 454 may use any load balancing scheme to select a request to be processed. In this example, arbitrator 454 may first process request 750 from a pre-fetch queue. In some embodiments, arbitrator 454 may process request 710 from an I / O request. In some embodiments, arbitrator 454 may process request 760 and request 770 from a pre-fetch queue. Therefore, the arbitrator may load memory to a memory medium, thereby allowing IOPS to be maximized.

[0066] Figure 8 A flow chart of prefetching data according to an example embodiment of the present disclosure is shown. For example, at block 810, according to an embodiment, a storage device may receive memory address information. For example, the storage device may receive prefetch memory address information from a host device. In some embodiments, the memory address information may be information related to application data used by the host. In some embodiments, the host may send other information that the storage device may use to determine the address of data to be loaded into the cache on the storage device. In some embodiments, the address information may correspond to data that may use any logic to determine the next data. In some embodiments, the address information may include one or more addresses. In some embodiments, the address information may be an indication of an address that the storage device may convert to an address on the storage device. In some embodiments, the storage device may use a table to convert address information on the storage device. In some embodiments, the data used to determine the address on the storage device may be sent by the host, an internal process, or the storage device itself.

[0067] At block 820, according to an embodiment, the storage device may store the address information in a buffer (e.g., a prefetcher queue). In some embodiments, the prefetcher may include a buffer. In some embodiments, the prefetcher may receive address information from the host and fill the buffer with the address information. In some embodiments, the buffer may be a ring buffer or some other queue to store the address information. In some embodiments, the storage device may load the address to be retrieved from the storage device. In some embodiments, the buffer may include a message from the host in the storage device. In some embodiments, the buffer may receive an indication of the address, which may be used to determine the real address. Although a first-out (FIFO) queue is described, in some embodiments, the prefetcher may be an ordered list that may be used to store address information on the storage device. In some embodiments, the buffer may contain other information for retrieving the address on the storage device.

[0068] At block 830, according to an embodiment, data can be loaded from the storage medium to the memory medium based on the memory address information. For example, if the buffer contains a memory address, the memory address can be used to load data from the storage medium to the memory medium. In some embodiments, the buffer can contain other information for determining the address information on the storage medium. For example, the buffer can contain an address range.

[0069] In some embodiments, the arbitrator may determine whether to retrieve address information from the prefetcher or process another read / write request. In some embodiments, the address information from the buffer may have the same format as the request from the host. In some embodiments, the arbitrator may know whether the request arriving at the arbitrator is from the host or from the prefetcher, and format the request in an appropriate format for the storage device to process the request. In some embodiments, the arbitrator may use any load balancing scheme to determine the order of the request from the host and the address information from the prefetcher. In some embodiments, the arbitrator may use AI to determine the order of the request. In some embodiments, the arbitrator may receive the request as a submission queue entry (SQE). In some embodiments, the arbitrator may forward a message from the prefetcher, and a controller on the storage device may process the address information to obtain an address on the storage medium. In some embodiments, the storage device may determine that the data has been loaded into the storage medium and process the next request. In some embodiments, the address information may be passed to the arbitrator, and the arbitrator may create a request to load the data into the queue. In some embodiments, any component of the storage device may create a request to load the data. For example, a cache may receive the address information, and the logic of the cache may allow it to load the data into the cache. In some embodiments, no programming changes may be required on the host, and the request from the arbitrator may be in the appropriate format for cache load data.

[0070] At block 840, according to an embodiment, the host may request data corresponding to the address information from the storage device, thereby allowing the host to retrieve data (e.g., cache hit) from the cache (at block 850), thereby increasing IOPS. In some embodiments, if the data is not in the cache, the data may be loaded into the cache and then retrieved by the host. In some embodiments, if the data is not in the cache, the data may be retrieved directly from the storage medium. In some embodiments, the storage device may also determine whether the address in the prefetcher is in use (e.g., whether the host will still use the data). In some embodiments, if the host does not need the data, the prefetcher may not pass the address information to the arbitrator. In some embodiments, the arbitrator may determine whether the data should be loaded into the cache. For example, if the data is already in the cache or is no longer needed, the arbitrator may not pass the request to the cache, but process the next request.

[0071] In some embodiments, the memory medium is accessible by software using load and / or store instructions, and the storage medium is accessible by software using read and / or write instructions.

[0072] In some embodiments, the memory medium may be accessed using a memory interface and / or protocol such as any generation of double data rate (DDR) (e.g., DDR4, DDR5, etc.), direct memory access (DMA), remote DMA (RDMA), open memory interface (OMI), compute express link (CXL), Gen-Z, etc., while the storage medium may be accessed using a storage interface and / or protocol such as serial ATA (SATA), small computer system interface (SCSI), serial attached SCSI (SAS), NVMe, NVMe-oF, etc.

[0073] Although some embodiments may be described in the context of memory media that may be implemented with memory media (such as DRAM), in other embodiments, other types of media (e.g., storage media) may be used for the memory media. For example, in some embodiments, some or all of the memory media 260 may be implemented with media other than memory media, which may have one or more relative characteristics (e.g., relative to the storage media 270) that may make one or both of them more suitable for their respective functions. For example, in some embodiments, the storage media 270 may have a relatively higher capacity, lower cost, etc., while some or all of the memory media 260 may have a relatively lower access latency, which may make them relatively more suitable for use as a cache.

[0074] Storage device 250, as well as any other devices disclosed herein, may be used in conjunction with one or more personal computers, smart phones, tablet computers, servers, server chassis, server racks, modems, data centers, edge data centers, mobile edge data centers, and / or any combination thereof.

[0075] Any functionality described herein, including any user functionality, device functionality, etc. (e.g., any control logic), may be implemented in hardware, software, firmware, or any combination thereof, including, for example, hardware and / or software combinational logic, sequential logic, timers, counters, registers, state machines, volatile memory (such as DRAM and / or SRAM), non-volatile memory (including flash memory), persistent memory (such as cross-grid non-volatile memory), memory with body resistance change, PCM, etc., and / or any combination thereof, complex programmable logic devices (CPLDs), FPGAs, ASICs, central processing units (CPUs) (including CISC processors such as x86 processors and / or RISC processors such as ARM processors), graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), data processing units (DPUs), etc., running instructions stored in any type of memory. In some embodiments, one or more components may be implemented as a system on a chip (SoC).

[0076] Some embodiments disclosed above have been described in the context of various implementation details, but the principles of the present disclosure are not limited to these or any other specific details. For example, some functions have been described as being implemented by certain components, but in other embodiments, functions may be distributed between different systems and components in different locations and with various user interfaces. Certain embodiments have been described as having specific processes, operations, etc., but these terms also cover embodiments in which specific processes, operations, etc. can be implemented with multiple processes, operations, etc., or embodiments in which multiple processes, operations, etc. can be integrated into a single process, step, etc. References to components or elements may refer only to a portion of a component or element. For example, a reference to a block may refer to an entire block or one or more sub-blocks. Terms such as "first" and "second" used in the present disclosure and claims may be used only for the purpose of distinguishing the elements they modify, and may not indicate any spatial or temporal order unless it is obvious from the context. In some embodiments, references to elements may refer to at least a portion of an element, for example, "based on" may refer to "based at least in part on", etc. References to a first element may not imply the presence of a second element. The principles disclosed herein have independent utility and may be embodied separately, and not every embodiment may utilize every principle. However, these principles can also be embodied in various combinations, some of which can amplify the benefits of the individual principles in a synergistic manner. The various details and embodiments described above can be combined to produce additional embodiments according to the inventive principles of this patent disclosure.

[0077] In some embodiments, a portion of an element may refer to less than or all of the element. A first portion of an element and a second portion of an element may refer to the same portion of an element. A first portion of an element and a second portion of an element may overlap (e.g., a portion of the first portion may be the same as a portion of the second portion).

[0078] Although certain exemplary embodiments have been described and shown in the accompanying drawings, it should be understood that these embodiments are merely illustrative, and the scope of the present disclosure is not limited to the embodiments described or shown herein. The present invention may be modified in arrangement and detail without departing from the concept of the present invention, and these changes and modifications are considered to fall within the scope of the appended claims.

Claims

1. A device comprising: Storage media; Storage media; buffer; as well as At least one circuit is configured to perform one or more operations, the one or more operations comprising: receiving memory address information; storing the memory address information in the buffer; determining that data can be loaded into the memory medium; and Data corresponding to the memory address information in the buffer is loaded from the storage medium to the memory medium.

2. The device according to claim 1, in, The memory address information is first memory address information; wherein the data is first data; and The at least one circuit is further configured to perform one or more operations, wherein the one or more operations include: receiving a memory access request including second memory address information; Determining to load second data based on the memory access request; and Based on the second memory address information, the second data is loaded from the storage medium to the memory medium.

3. The device according to claim 1, wherein: Loading data includes: obtaining at least one address from the memory address information; and The at least one address is added to the buffer.

4. The device according to claim 1, wherein: The memory address information includes an indication of a length of the address and at least one address.

5. The device according to claim 1, wherein: The memory address information includes at least one address and an indication of a number of addresses.

6. The device according to claim 1, wherein: The at least one circuit is further configured to perform one or more operations, the one or more operations comprising: receiving an instruction to make the buffer available; and The buffer is made available.

7. The device according to claim 1, wherein: The at least one circuit is further configured to perform one or more operations, the one or more operations comprising: receiving an instruction to make the memory medium available; and The storage medium is made available.

8. The device according to claim 1, wherein: The at least one circuit includes a control register configured to perform one or more operations including: Based on the control register, determining a state of the buffer; and Based on the status, operation of the at least one circuit is modified.

9. A method for memory loading, comprising: receiving memory address information; storing the memory address information in a buffer; determining that data can be loaded into the storage medium; as well as Based on the memory address information in the buffer, data is loaded from a storage medium to the memory medium.

10. The method according to claim 9, in, The memory address information is first memory address information; wherein the data is first data; and Wherein, the method further comprises: receiving a memory access request including second memory address information; Determining to load second data based on the memory access request; and Based on the second memory address information, the second data is loaded from the storage medium to the memory medium.

11. The method according to claim 9, wherein: Loading data includes: obtaining at least one address from the memory address information; and The at least one address is added to the buffer.

12. The method according to claim 9, wherein: The memory address information includes an indication of a length of the address and at least one address.

13. The method according to claim 9, wherein: The memory address information includes at least one address and an indication of a number of addresses.

14. The method according to claim 9, further comprising: receiving an instruction to make the buffer available; as well as The buffer is made available.

15. The method according to claim 9, further comprising: determining a state of the buffer; as well as Based on the status, operation of the buffer is modified.

16. A system for memory loading, comprising: A host device, comprising an application and a thread, wherein the thread comprises memory address information of the application; and Storage devices, including: Storage media; Storage media; Buffer; and At least one circuit is configured to perform one or more operations, the one or more operations comprising: Receiving the memory address information; storing the memory address information in the buffer; determining that data can be loaded into the memory medium; and Based on the memory address information in the buffer, data is loaded from the storage medium to the memory medium.

17. The system according to claim 16, in, The memory address information is first memory address information; wherein the data is first data; and The at least one circuit is further configured to perform one or more operations, wherein the one or more operations include: receiving a memory access request including second memory address information; Determining to load second data based on the memory access request; and Based on the second memory address information, the second data is loaded from the storage medium to the memory medium.

18. The system of claim 16, wherein: The memory address information includes an indication of a length of the address and at least one address.

19. The system of claim 16, wherein: The memory address information includes at least one address and an indication of a number of addresses.

20. The system of claim 16, wherein: The at least one circuit is further configured to perform one or more operations, the one or more operations comprising: receiving an instruction to make the buffer available; and The buffer is made available.

Images