Systems, methods, and apparatus for in-store machine learning operations

Abstract

Systems, methods, and apparatus are provided for in-store machine learning operations. In some embodiments, the system includes a first persistent memory and a control and inference circuit. The first persistent memory may be connected to a control and inference circuit through a broadband data connection, and the control and inference circuit may be configured to perform arithmetic operations.

Description

[0001] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 531,732, filed on August 9, 2023, entitled “SYSTEM AND APPARATUS FOR IN-FLASH INFERENCE OF LARGE AI MODELS,” and U.S. Application No. 18 / 475,141, filed on September 26, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] One or more aspects of embodiments according to the present disclosure relate to machine learning, and more particularly, to systems and methods for in-memory inference. Background Art

[0003] Machine learning can be useful for a variety of classification and generation operations. Inference operations in large machine learning models can use large amounts of data and large amounts of processing resources.

[0004] Aspects of the present disclosure are related to this general technical environment. Summary of the invention

[0005] According to an embodiment of the present disclosure, a system is provided, comprising: a first persistent memory; and a control and inference circuit, wherein: the first persistent memory is connected to the control and inference circuit via a broadband data connection; and the control and inference circuit is configured to perform arithmetic operations.

[0006] In some embodiments: the control and inference circuit includes a first persistent memory controller; the system also includes: an interface circuit, a second persistent memory, and a second persistent memory controller, the second persistent memory is connected to the second persistent memory controller; the control and inference circuit is connected to the interface circuit; and the second persistent memory controller is connected to the interface circuit.

[0007] In some embodiments, the system further comprises: an interface circuit, wherein: the interface circuit comprises a serial interface; and the control and inference circuit is connected to the interface circuit.

[0008] In some embodiments, the system further comprises: an interface circuit; a second persistent memory; and a second persistent memory controller, wherein: the control and inference circuit comprises the first persistent memory controller; the interface circuit comprises a serial interface; the control and inference circuit is connected to the interface circuit; and the second persistent memory is connected to the interface circuit.

[0009] In some embodiments, the system also includes: a second persistent memory, and a second persistent memory controller, wherein: the control and inference circuit includes a first persistent memory controller, the first persistent memory includes flash memory, the second persistent memory includes flash memory, the first persistent memory controller includes a first flash memory controller, and the second persistent memory controller includes a second flash memory controller.

[0010] In some embodiments, the broadband data connection comprises a serial connection.

[0011] In some embodiments, the system further includes: a first interface circuit, and an artificial intelligence accelerator, the artificial intelligence accelerator including an artificial intelligence processing circuit and a second interface circuit, wherein: the first interface circuit is connected to the second interface circuit, and the artificial intelligence accelerator is configured to perform inference operations of the neural network with the help of a first persistent memory and a control and inference circuit.

[0012] In some embodiments, the system further comprises: a first interface circuit, and an artificial intelligence accelerator connected to the first interface circuit, wherein: the artificial intelligence accelerator is configured to perform inference operations of the neural network with the help of the first persistent memory and the control and inference circuit; and the first persistent memory and the control and inference circuit are configured to perform operations selected from the group consisting of pruning, sparsification, compression, quantization, and approximation.

[0013] In some embodiments, the first persistent memory is part of a first semiconductor die; and the control and inference circuitry is part of a second semiconductor die.

[0014] In some embodiments, the first persistent memory is part of a first semiconductor die; the control and inference circuitry is part of a second semiconductor die; and the first semiconductor die and the second semiconductor die are part of a stack of dies.

[0015] In some embodiments, the system includes: a random access memory, wherein: the first persistent memory is part of a first semiconductor die; the control and inference circuitry is part of a second semiconductor die; the random access memory is part of a third semiconductor die; and the first semiconductor die, the second semiconductor die, and the third semiconductor die are part of a stack of dies.

[0016] In some embodiments, the system includes: a random access memory, wherein: the first persistent memory is part of a first semiconductor die; the control and inference circuitry is part of a second semiconductor die; the random access memory is part of a third semiconductor die; and the first semiconductor die is stacked on the second semiconductor die and the third semiconductor die.

[0017] In some embodiments, the system includes: a random access memory, wherein: the control and inference circuit includes: a persistent memory controller; and a multiplication-accumulation circuit, the first persistent memory is part of a first semiconductor die; the persistent memory controller is part of a second semiconductor die; the multiplication-accumulation circuit is part of a third semiconductor die; the random access memory is part of a fourth semiconductor die; and the first semiconductor die is stacked on the second semiconductor die, the third semiconductor die, and the fourth semiconductor die.

[0018] According to an embodiment of the present disclosure, a method is provided, the method comprising: performing an inference operation of a neural network, the execution steps comprising: reading weights from a persistent memory into a random access memory; multiplying the weights by elements of an input feature map to form a first product; and calculating activation based on the first product, wherein the step of reading the weights from the persistent memory into the random access memory comprises: reading the weights from the persistent memory into the random access memory via a broadband data connection.

[0019] In some embodiments, the method further comprises storing the activation in a random access memory.

[0020] In some embodiments: the step of performing the inference operation includes performing the inference operation in a system, wherein the system includes: a persistent memory, and a control and inference circuit; the control and inference circuit is connected to the persistent memory via a broadband data connection; and the control and inference circuit includes: a persistent memory controller; and a multiply-accumulate circuit.

[0021] According to an embodiment of the present disclosure, a device is provided, comprising: a connector; a first persistent memory; and a control and inference circuit, wherein: the first persistent memory is connected to the control and inference circuit via a broadband data connection; and the connector is suitable for connecting the device to a mobile computing device.

[0022] In some embodiments: the control and inference circuit includes a first persistent memory controller; the device also includes: an interface circuit, a second persistent memory, and a second persistent memory controller, the second persistent memory is connected to the second persistent memory controller; the control and inference circuit is connected to the interface circuit; and the second persistent memory controller is connected to the interface circuit.

[0023] In some embodiments: the first persistent memory and the control and inference circuitry are configured to: perform an operation selected from the group consisting of pruning, sparsification, compression, quantization, and approximation.

[0024] In some embodiments: the first persistent memory is part of a first semiconductor die; and the control and inference circuitry is part of a second semiconductor die. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] These and other features and advantages of the present disclosure will be appreciated and understood with reference to the specification, claims and drawings, in which: Figure 1A is a block diagram of a host and a storage device according to an embodiment of the present disclosure; Figure 1B is a block diagram of a storage device according to an embodiment of the present disclosure; Figure 1C is a system-level block diagram according to an embodiment of the present disclosure; Figure 1D is a system-level diagram of a host and a persistent storage device with inference capabilities according to an embodiment of the present disclosure; Figure 1E is a schematic perspective view of a stacked package of dies according to an embodiment of the present disclosure; Figure 2 is a block diagram of an artificial intelligence accelerator and a persistent storage device with inference capabilities according to an embodiment of the present disclosure; Figure 3 is a system-level diagram of a portable computing device and a persistent storage device with inference capabilities according to an embodiment of the present disclosure; and Figure 4 is a flowchart of a method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of exemplary embodiments of systems and methods for storing inferences according to the present disclosure, and is not intended to represent the only form in which the present disclosure may be constructed or utilized. The description sets forth the features of the present disclosure in conjunction with the illustrated embodiments. However, it will be understood that the same or equivalent functions and structures may be implemented by different embodiments that are also intended to be included within the scope of the disclosure. As shown elsewhere herein, the same element numbers are intended to indicate the same elements or features.

[0027] Machine learning operations (such as neural network inference operations) may involve calculating the product of weights and elements of an input feature map. In a large neural network, a large number of such products may be calculated for each inference operation, or for each layer of the neural network for each inference operation. The weights may be stored in a persistent storage device (discussed in further detail below). The persistent storage device may be connected to a host. The host may include processing circuitry (e.g., a central processing unit (CPU) and main memory or system memory). In operation, the host may receive elements of an input feature map as input, or the host (e.g., a processing circuit of the host) may calculate an output feature map of one layer, and the elements of the input feature map of the next layer may be elements of the calculated output feature map.

[0028] The host may read the weights from the persistent storage device into the system memory, and the processing circuitry of the host may perform the calculation of the product of the weights and the elements of the input feature map. This process may be repeated for each of potentially many layers of the neural network.

[0029] The process of reading weights from persistent storage can consume a large amount of power and can also cause a large amount of delay due to the limited latency and bandwidth of the connection between the host and the persistent storage. In addition, the calculation of weights in the host can consume power in the processing circuitry and in the system memory, and the calculation of weights in the host can consume cycles of the processing circuitry and memory that could otherwise be used for other processing tasks. Therefore, such use of the processing circuitry of the host and the system memory of the host can result in degradation of system performance.

[0030] Thus, in some embodiments, machine learning operations (e.g., neural network inference operations) may be performed in a persistent storage device. Inference operations of a neural network may involve computing products of (i) weights and (ii) elements of an input feature map, summing such products, and computing an activation function based on such sums. In some embodiments, weights of a layer of a neural network or all weights of an entire neural network may be stored in a persistent memory of a persistent storage device. The persistent storage device may also have dedicated circuitry for performing neural network inference operations.

[0031] For example, the persistent storage device may include a persistent memory (which may store weights), a random access memory (RAM) (e.g., a static random access memory), and a multiply-accumulate circuit. In operation, the persistent storage device may receive elements of an input feature map (e.g., from a host connected to the persistent storage device or from an artificial intelligence accelerator connected to the persistent storage device); the persistent storage device may store these elements of the input feature map in the random access memory. The persistent storage device may then calculate an output value (e.g., a sum of products of the weights and elements of the input feature map, or an activation value produced by evaluating an activation function for such a sum of products) (using the weights stored in the persistent storage and the processing power of the multiply-accumulate circuit) and return the output value (e.g., to the host or to the artificial intelligence accelerator). In some embodiments, the persistent storage device may have a dongle form factor and may be configured to be inserted into a portable computing device to enhance the neural network inference capabilities of the portable computing device.

[0032] Figure 1A A system, which may be referred to as server 100, is shown according to some example embodiments disclosed. Figure 1A, the server 100 may include a host device (or simply referred to as a "host") 102 and a storage device 104 (which may be a persistent storage device 104). In some embodiments, the host device 102 may be located together with the persistent storage device 104, while in other embodiments, the host device 102 may be separated from the persistent storage device 104. The host device 102 may include any suitable computing device connected to the persistent storage device 104 (such as, for example, a personal computer (PC), a portable electronic device, a handheld device, a laptop computer, etc.).

[0033] The host device 102 may be connected to the persistent storage device 104 through the host interface 106. The host device 102 may issue a data request command or an input-output (IO) command (eg, a read command or a write command) to the persistent storage device 104 through the host interface 106, and may receive a response from the persistent storage device 104 through the host interface 106.

[0034] Host device 102 may include host processor 108 and host memory 110. Host processor 108 may be a processing circuit (discussed in further detail below) (eg, such as a general purpose processor or central processing unit (CPU) core of host device 102).

[0035] The host processor 108 may be connected to other components via an address bus, a control bus, a data bus, etc. The host memory 110 may be considered a high-performance main memory (e.g., primary memory) of the host device 102. For example, in some embodiments, the host memory 110 may include (or may be) a volatile memory (e.g., such as a dynamic random access memory (DRAM)). However, the present disclosure is not limited thereto, and the host memory 110 may include (or may be) any suitable high-performance main memory (e.g., primary memory) replacement for the host device 102 known to those skilled in the art. For example, in other embodiments, the host memory 110 may be a relatively high-performance non-volatile memory (e.g., NAND flash memory, phase change memory (PCM), resistive RAM, spin transfer torque RAM (STTRAM), any suitable memory based on PCM technology, memristor technology, or resistive random access memory (ReRAM)), and may include, for example, chalcogenide, etc.

[0036] Persistent storage 104 may operate as secondary storage that may persistently store data accessible by host device 102. In this context, persistent storage 104 may comprise relatively slow memory when compared to the high performance memory of host memory 110.

[0037] For example, in some embodiments, the persistent storage device 104 may be a secondary storage (e.g., such as a solid-state drive (SSD)) of the host device 102. However, the present disclosure is not limited thereto, and in other embodiments, the storage device 104 may include (or may be) any suitable storage device (such as, for example, a magnetic storage device (e.g., a hard disk drive (HDD)), an optical storage device (e.g., a Blu-ray drive, a compact disk (CD) drive, a digital versatile disk (DVD) drive, etc.), other kinds of flash memory devices (e.g., a USB flash drive, etc.), etc.).

[0038] In various embodiments, persistent storage device 104 may conform to a large form factor standard (e.g., a 3.5-inch hard drive form factor), a small form factor standard (e.g., a 2.5-inch hard drive form factor), an M.2 form factor, an E1.S form factor, etc. In other embodiments, persistent storage device 104 may conform to any suitable or desired derivative of these form factors. For convenience, hereinafter, persistent storage device 104 may be described in the context of a solid-state drive, but the present disclosure is not limited thereto.

[0039] Persistent storage device 104 may be communicatively connected to host device 102 via host interface 106. Host interface 106 may facilitate communication between host device 102 and persistent storage device 104 (e.g., using a connector and protocol). In some embodiments, host interface 106 may, for example, facilitate the exchange of storage requests (or "commands") and responses (e.g., command responses) between host device 102 and persistent storage device 104. In some embodiments, host interface 106 may facilitate the transfer of data to and from host memory 110 of host device 102 through persistent storage device 104.

[0040] For example, in various embodiments, the host interface 106 (e.g., the connector and the connector's protocol) may include (or may conform to) Small Computer System Interface (SCSI), Non-Volatile Memory Express (NVMe), Peripheral Component Interconnect Express (PCIe), Remote Direct Memory Access (RDMA) over Ethernet, Serial Advanced Technology Attachment (SATA), Fibre Channel, Serial Attached SCSI (SAS), NVMe over Fabric (NVMe-oF), etc. In other embodiments, the host interface 106 (e.g., the connector and the connector's protocol) may include (or may conform to) various general-purpose interfaces (e.g., such as Ethernet, Universal Serial Bus (USB), etc.).

[0041] In some embodiments, persistent storage 104 may include a persistent memory controller (or "storage controller") 112, storage memory 114 (which may also be referred to as a buffer), non-volatile memory (NVM) 116, and storage interface 118. Storage memory 114 may be a high-performance memory of persistent storage 104, and may include (or may be) a volatile memory (e.g., such as DRAM), but the present disclosure is not limited thereto, and storage memory 114 may be, for example, any suitable type of high-performance volatile memory or non-volatile memory.

[0042] The non-volatile memory 116 may persistently store data received, for example, from the host device 102. The non-volatile memory 116 may include, for example, NAND flash memory, but the disclosure is not limited thereto and the non-volatile memory 116 may include any suitable type of memory (e.g., magnetic disks, tapes, optical disks, etc.) for persistently storing data depending on the implementation of the persistent storage device 104.

[0043] The persistent memory controller 112 may be connected to the non-volatile memory 116 through a storage interface 118. In the context of an SSD, the storage interface 118 may be referred to as a flash channel and may be an interface by which the non-volatile memory 116 (e.g., NAND flash memory) may communicate with a processing component (e.g., the persistent memory controller 112) or other devices. Commands (such as reset, write enable, control signals, clock signals, etc.) may be sent through the storage interface 118.

[0044] In some embodiments, the software interface may be used in combination with hardware elements that may be used to test or verify the operation of the storage interface 118. The software may be used to read data from and write data to the non-volatile memory 116 via the storage interface 118. In some embodiments, the software may include firmware (e.g., to control write operations, erase operations, and read operations) that may be downloaded onto the hardware elements.

[0045] A persistent memory controller 112, which may be a processing circuit (discussed in further detail below), may be connected to the host interface 106 and may manage signaling, for example, through the host interface 106. In some embodiments, the persistent memory controller 112 may include an associated software layer (e.g., a host interface layer) to manage the physical connectors of the host interface 106. The persistent memory controller 112 may respond to input requests or output requests received, for example, from the host device 102 through the host interface 106. The persistent memory controller 112 may also manage the storage interface 118, for example, to control the non-volatile memory 116 and provide access to and from the non-volatile memory 116.

[0046] For example, the persistent memory controller 112 may include at least one processing component embedded in the persistent memory controller 112 for interfacing with the host device 102 and the non-volatile memory 116. The processing component may include, for example, a general-purpose digital circuit (e.g., a microcontroller, a microprocessor, a digital signal processor, or a logic device (e.g., a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.)) that is "capable of executing (e.g., via firmware or software) data access instructions to provide access to data stored in the non-volatile memory 116 according to the data access instructions." For example, the data access instructions may correspond to data request commands and may include any suitable data storage and retrieval algorithm (e.g., read, write, or erase) instructions, etc.

[0047] Figure 1B is a block diagram of a persistent storage device 104 (eg, a solid state drive) according to some example embodiments disclosed. A host interface 106 is used by a host 102 to communicate with the persistent storage device 104 .

[0048] Data write and read input and output commands, along with various media management commands (such as non-volatile memory express (NVMe) identification commands and NVMe grab log commands), may be received by the persistent storage device 104 through the host interface 106. The host interface 106 may also be used by the persistent storage device 104 to perform data transfers to and from the host system memory. The persistent storage device 104 may store data in a non-volatile memory 116 (e.g., non-and (NAND) flash memory) (e.g., in a memory die (or referred to as a die or persistent memory) 117 containing memory cells, each of which may be (as discussed above) a single-level cell (SLC), a multi-level cell (MLC), or a triple-level cell (TLC), for example). A flash translation layer (FTL) that may be implemented in the persistent memory controller 112 (e.g., based on firmware (e.g., based on firmware stored in the non-volatile memory 116)) may provide a mapping between a logical address used by the host and a physical address of data in the non-volatile memory 116.

[0049] The persistent storage device 104 may also include (i) a buffer (e.g., storage memory 114) (which may include (e.g., be composed of) dynamic random access memory (DRAM), and (ii) a flash interface (or "flash controller") 121 for providing appropriate signals to the memory die 117 of the non-volatile memory 116. Some or all of the host interface 106, the flash translation layer (as discussed above), the storage memory 114 (e.g., the buffer), and the flash interface 121 may be implemented in a processing circuit, which may be referred to as a persistent storage device controller 112 (or simply persistent memory controller 112).

[0050] NAND flash memory can be read or written at a granularity of a flash page that can be between 8kB and 16kB. Before a flash page is reprogrammed with new data, the flash page can first be erased. The granularity of the erase operation can be one NAND block or "physical block" that can include, for example, between 128 pages and 256 pages.

[0051] Because the granularity of erase operations and program operations is different, garbage collection (GC) can be used to free up partially invalid physical blocks and make room for new data. The garbage collection operation can (i) identify fragmented flash blocks where a large portion (e.g., most) of the pages are invalid, and (ii) erase each such physical block. When garbage collection is complete, the pages in the erased physical blocks can be recycled and added to a free list in the flash translation layer.

[0052] The non-volatile memory 116 (e.g., if the non-volatile memory 116 includes or is flash memory) may be capable of being programmed and erased only a limited number of times. This may be referred to as the maximum number of program / erase cycles (P / E cycles) that the non-volatile memory 116 can sustain. In order to maximize the life of the persistent storage 104, the persistent storage controller 112 may try to distribute write operations across all physical blocks of the non-volatile memory 116; this process may be referred to as wear leveling.

[0053] A mechanism that may be referred to as "read disturbance" may reduce persistent storage device reliability. A read operation on a NAND flash memory cell may, for example, cause the threshold voltage of a nearby, unread flash memory cell in the same physical block to change. Such disturbance may change the logical state of the unread cell and may result in uncorrectable error correction code (ECC) read errors, reducing flash memory endurance.

[0054] To avoid such an outcome, the flash translation layer may have a counter of the total number of reads to the physical block since the last erase operation. The contents of the physical block may be copied to a new physical block, and when the counter exceeds a threshold (e.g., 50,000 reads for a multi-level cell), the physical block may be recycled to avoid unrecoverable read disturb errors. As an alternative, in some embodiments, test reads may be periodically performed within the physical block to check the error correction code error rate; if the error rate approaches the error correction code capability, the data may be copied to a new physical block.

[0055] Because of the relocation of data performed by various operations (e.g., garbage collection) in persistent storage 104, the amount of data that is erased and rewritten may be greater than the amount of data written to persistent storage 104 by the host. Each time data is relocated without being changed by the host system, an amount known as write amplification increases, and the life of non-volatile memory 116 may be reduced. Write amplification may be measured as the ratio of (i) the number of writes submitted to the flash memory to (ii) the number of writes from the host system.

[0056] Figure 1C is a system-level diagram of a system capable of performing in-memory machine learning operations according to some example embodiments disclosed. Figure 1C As shown in FIG. 1 , multiple servers 100 are connected to each other via a network (eg, a portable computing device) 105. Within each server 100, a host 102 is connected to a persistent storage device 104 (which may be, for example, a solid state drive (SSD)). The persistent storage device 104 may (as in FIG. Figure 1A and Figure 1B ) has a form factor suitable for any one of a plurality of form factors for persistent storage devices, including but not limited to: a 2.5" form factor, a 1.8" form factor, a MO-297 form factor, a MO-300 form factor, an M.2 form factor, and an Enterprise and Data Center SSD Form Factor (EDSFF).

[0057] Persistent storage device 104 may have an electrical interface (which may be referred to as a "host interface") 106 through which persistent storage device 104 may be connected to host 102 (e.g., Figure 1A and Figure 1BThe electrical interface 106 may be any one of a plurality of interfaces suitable for a persistent storage device, including a peripheral component interconnect (PCI), PCI Express (PCIe), Ethernet, a small computer system interface (SCSI), a serial AT attachment (SATA), and a serial attached SCSI (SAS), or a universal flash storage (UFS). The universal flash storage may include a plurality of serial interfaces, each of which may include a full-duplex high-speed serial path. The persistent storage device 104 may include an interface circuit that "operates as an interface adapter between the host interface 106 and one or more internal interfaces in the persistent storage device 104."

[0058] As used herein, "persistent memory" refers to non-volatile memory (e.g., persistent memory is non-volatile memory that can continue to store data when power is not supplied to the persistent memory). Persistent storage device 104 may include interface circuit 130 (or interface 130) ( Figure 1D ), interface circuit 130 ( Figure 1D ) operates as an interface adapter between a host interface and one or more internal interfaces in persistent storage device 104.

[0059] The host interface may be used by the host 102 to communicate with the persistent storage device 104, for example, by sending write commands and read commands, which may be received by the persistent storage device 104 through the host interface 106. In some embodiments, the host 102 may send elements of the input feature map to the persistent storage device 104 through the host interface 106, and the persistent storage device 104 may calculate the product of the elements of the input feature map and the weights. The host interface may also be used by the persistent storage device 104 to perform data transfers to and from the system memory of the host 102. Such data transfers may be performed using direct memory access (DMA). For example, when the host 102 sends a write command to the persistent storage device 104, the persistent storage device 104 may use direct memory access to obtain data to be written to the non-volatile memory 116 from the host memory 110 of the host device 102, and the persistent storage device 104 may then save the obtained data to the non-volatile memory 116.

[0060] Similarly, if the host 102 sends a read command to the persistent storage device 104, the persistent storage device 104 may read the requested data (i.e., the data specified in the read command) from the non-volatile memory 116 and save the requested data in the host memory 110 of the host device 102 using direct memory access. The persistent storage device 104 may store the data in a persistent memory (e.g., NAND flash memory) (e.g., in a memory die including memory cells, each of which may be, for example, a single-level cell (SLC), a multi-level cell (MLC), or a triple-level cell (TLC)).

[0061] Persistent storage device 104 may (e.g., Figure 1A and Figure 1B In an embodiment of the present invention, the data is stored in a persistent memory (e.g., NAND flash memory) (e.g., stored in a memory die including memory cells, each of which can be, for example, a single-level cell (SLC), a multi-level cell (MLC), a triple-level cell (TLC), etc.).

[0062] A flash translation layer (FTL) of persistent storage device 104 may provide a mapping between logical addresses used by host 102 and physical addresses of data in the persistent storage. Persistent storage device 104 may also include (i) a buffer (which may include (e.g., consist of) dynamic random access memory (DRAM)), and (ii) a persistent memory controller (e.g., a flash memory controller) for providing appropriate signals to the persistent storage. Some or all of the host interface, flash translation layer, buffer, and persistent memory controller may be implemented in a processing circuit that may be referred to as a persistent storage device controller.

[0063] Figure 1C The system may be employed in various ways to perform various computations including inference computations of neural networks. To perform inference computations of neural networks (e.g., a host 102 of one of the servers 100 may read the weights of a layer of the neural network from the persistent storage device 104, and the host 102 may multiply each weight by a corresponding element of an input feature map to form a corresponding product, and the host 102 may sum the products). The host 102 may then calculate activations based on the summation.

[0064] In some embodiments, some or all of the computations performed by host 102 in some embodiments may be performed at least in part by persistent storage 104. For example, as shown, persistent storage 104 may include non-volatile memory 116 and control and processing circuitry, which may be referred to as control and inference circuitry 120. Control and inference circuitry 120 may be, for example, in an integrated circuit (or, as shown (e.g., in a plurality of integrated circuits). Figure 1D The control and inference circuitry 120 may include a persistent memory controller 112 (in a separate integrated circuit) and may also include an inference processing circuit 122. The control and inference circuitry 120 may be a processing circuit, or the control and inference circuitry 120 may include one or more processing circuits (discussed in further detail below).

[0065] In some embodiments, the persistent memory controller of the control and inference circuitry 120 and the inference processing circuitry of the control and inference circuitry 120 may be fabricated as separate integrated circuits (e.g., on separate semiconductor dies). These dies may be connected together by suitable interconnects using, for example, wire bonds or traces as conductors in a redistribution layer. The connector may be a plurality of serial connectors having a plurality of serial transceivers (each of the transceivers may conform to a suitable serial interface standard, such as, for example, an ultra short reach (XSR) or an extremely short reach (USR)).

[0066] The non-volatile memory 116 and the control and inference circuitry 120 may be connected via a broadband data connection 125. As used herein, a "broadband data connection" may include a data connection having a bandwidth of a predetermined threshold (e.g., at least 5 gigabits per second). In some embodiments, the broadband data connection 125 has a bandwidth between 5 gigabits per second and 100 gigabits per second. The broadband data connection 125 may be implemented using a peripheral component interconnect express (PCIe) (with a sufficient number of lanes (each of which may be or include a serial connection) to achieve the desired bandwidth) or with a serial connection (e.g., with multiple lanes) that conforms to another serial connection standard (such as medium reach (MR) or very short reach (VSR)). In some embodiments, the non-volatile memory 116 is large enough to store an entire model (e.g., a Generative Pre-Trained Transformer (GPT)) for a large neural network (e.g., the non-volatile memory 116 may store all weights for such a model). In some embodiments, non-volatile memory 116 has a capacity of a predetermined threshold (eg, at least 1 Gigabyte).

[0067] The inference processing circuitry of the control and inference circuitry 120 may include circuitry configured to perform arithmetic operations (e.g., multiplication-accumulation circuitry configured to efficiently perform multiplications of weights and elements of an input feature map to form products and accumulate (e.g., sum) the products). The control and inference circuitry 120 may include a random access memory 123 ( Figure 1E ), random access memory 123 ( Figure 1E ) may be part of the inference processing circuitry of control and inference circuitry 120, or may be a separate element of control and inference circuitry 120. Random access memory 123 may be, for example, static random access memory (SRAM) or dynamic random access memory (DRAM).

[0068] In operation, when commanded by the host 102 to perform an inference operation or a portion of an inference operation, the control and inference circuitry 120 may read weights (e.g., a portion of weights corresponding to a layer of a neural network, or all weights corresponding to a layer of a neural network) from the non-volatile memory 116 to the random access memory 123, and the multiplication-accumulation circuitry may perform a multiplication and accumulation operation (e.g., a multiplication and summation (e.g., accumulation) operation for the inference operation).

[0069] Using broadband data connection 125 to read weights from non-volatile memory 116 may enable persistent storage 104 to perform inference operations significantly faster than if the inference operations were performed in host 102 after the weights were read from persistent storage 104 by host 102. In some embodiments, the operations that control and inference circuitry 120 is capable of performing include inference operations that support operations such as automatic speech recognition, natural language processing, and text-to-speech.

[0070] As mentioned above, persistent storage device 104 may include interface circuitry 130 that is operable to provide a host interface to host 102. Interface circuitry 130 may operate as an interface adapter between host 102 and control and inference circuitry 120, to which interface circuitry 130 may be connected via a data connection conforming to any standard suitable for internal connections within persistent storage device 104, such as Peripheral Component Interconnect Express (PCIe) or another serial connection standard such as Medium Reach (MR) or Very Short Reach (VSR).

[0071] like Figure 1DAs shown in (which shows a block diagram of a persistent storage device with inference capabilities according to some example embodiments disclosed), the persistent storage device 104 may also include additional or "auxiliary" persistent storage (e.g., NVM) 119 and an auxiliary persistent storage controller 135, which together with the interface circuit 130 can enable the persistent storage device 104 to provide persistent storage (e.g., without in-device processing) to the host 102. Such a capability can be used, for example, to store user data (e.g., image files or sound files) or applications, or the operating system of the host 102.

[0072] Reference Figure 1E (which shows a schematic perspective view of a stacked package of dies according to some example embodiments disclosed), non-volatile memory 116, persistent memory controller 112, inference processing circuit 122, and random access memory 123 may be fabricated on separate semiconductor dies, and they may be packaged in a stacked (three-dimensional) package, where, for example, non-volatile memory 116 is stacked on (i) persistent memory controller 112, (ii) inference processing circuit 122, and (iii) random access memory 123. The stacked package may make it feasible to provide a large number of conductive connections 124 (only a small number of conductive connections are shown for ease of illustration), a portion of which may be used to implement broadband data connections 125. The stack of dies may be packaged in a system-in-package (SiP) or a multi-chip package (MCP). However, embodiments are not limited thereto. For example, in one embodiment, non-volatile memory 116 may be part of a first semiconductor die, persistent memory controller 112 and inference processing circuit 122 may be part of a second semiconductor die, and random access memory 123 may be part of a third semiconductor die. In this case, the first semiconductor die, the second semiconductor die, and the third semiconductor die may be part of a stack of dies, and / or the first semiconductor die may be stacked on the second semiconductor die and the third semiconductor die.

[0073] Figure 2 An embodiment is shown in which the persistent storage device 104 is connected to an artificial intelligence (AI) accelerator 205 according to some example embodiments disclosed. The artificial intelligence accelerator 205 may include an artificial intelligence engine 210 (or "artificial intelligence processing circuit") and an accelerator interface circuit 215 (or interface 215). In operation, the artificial intelligence accelerator 205 may supervise and manage the execution of inference operations, delegating operations to the persistent storage device 104 when delegation results in faster (or lower power) operations.

[0074] For example, the control and inference circuitry 120 of the persistent storage device 104 may be configured to efficiently perform multiplications and accumulations as mentioned above, but the control and inference circuitry 120 may not have hardware capable of evaluating an activation function (which may, for example, require computing an exponential function) in a particular amount of time or for an amount of energy consumed compared to that used by a more general purpose floating point processor (e.g., a CPU with a floating point accelerator). In some embodiments, the artificial intelligence accelerator 205 (e.g., the artificial intelligence engine 210 of the artificial intelligence accelerator 205) may include (i) general purpose computational hardware (e.g., a floating point processor) or (ii) specialized hardware (such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a coprocessor) for performing such computations (e.g., computation of activation functions).

[0075] Thus, in such an embodiment, the artificial intelligence accelerator 205 can instruct the persistent storage device 104 to calculate the sum of the products of the weight and the elements of the input feature map for each artificial neuron of the layer for each layer of the neural network, and return the sum to the artificial intelligence accelerator 205, and then the artificial intelligence accelerator 205 can calculate the activation of each artificial neuron of the layer. Then, this group of activations can become the elements of the input feature map of the next layer of the neural network.

[0076] In some embodiments, the number of stacks (eg, stacked dies) up to a predetermined threshold (eg, 16 stacks (eg, stacked dies)) may be sufficient to provide a plurality of stacks. Figure 1E ) can be included in a package, resulting in a broadband data connection 125 with a bandwidth for sequential read operations up to a predetermined threshold (e.g., 300 gigabytes per second). In some embodiments, the persistent storage device 104 is capable of performing (i) pruning (skipping products where one or two factors are less than a threshold), sparsification (skipping products where one or two factors are zero), compression (encoding using an encoding method that reduces the storage space occupied (e.g., entropy codes)), quantization (mapping values ​​to a set of more coarsely spaced discrete values), or approximation (calculating approximated products when doing so can be done faster or using less energy) in the control and inference circuitry 120, or (ii) offloading of neural network layers or operations (e.g., multi-head attention, a processing method involving multiple parallel passes through an attention mechanism). In some embodiments, the persistent storage device 104 is capable of performing an entire inference operation or a portion of an inference operation (e.g., one layer of a neural network at a time).

[0077] In some embodiments, Figure 3As shown in , which shows a system-level diagram of a portable computing device and a persistent storage device with inference capabilities according to some example embodiments disclosed, a persistent storage device 104 is configured to be connected to a portable computing device 105 (e.g., a laptop computer, a tablet computer, or a mobile phone). For example, the persistent storage device 104 may have a form factor of a dongle (e.g., having a universal serial bus (USB) port) connector. Such an embodiment may allow a laptop computer, tablet computer, or mobile phone that may not have sufficient processing power to perform inference operations of large neural networks in an acceptable amount of time to greatly speed up the ability of the laptop computer, tablet computer, or mobile phone to perform such inference operations by offloading a portion (e.g., one layer at a time) or all of the inference operations to the persistent storage device 104.

[0078] Some embodiments are generally applicable to any (existing or future) hardware device as a built-in chip or dongle (or other plug-in form factor (such as a card with a suitable connector)). The broadband data connection 125 of the persistent storage device 104 can be used to read the artificial intelligence model from the non-volatile memory 116 to the control and inference circuit 120. In embodiments where a dongle form factor is adopted, input information (such as a user language query (which can be a sound file containing spoken words) or an input image) can be transmitted to the dongle via a universal serial bus (or other suitable connection), and only the output results (e.g., the output from the generative pre-trained transformer or other inference output) can be sent back to the host 102 (e.g., to a portable computing device).

[0079] In some embodiments, the workflow may include (i) feeding inputs (e.g., elements of an input feature map) to random access memory 123 from the outside (e.g., from host 102 or from artificial intelligence accelerator 205), (ii) reading a portion of weights from non-volatile memory 116 to random access memory 123 via broadband data connection 125, (iii) performing inference operations (e.g., multiplications and accumulations) by control and inference circuitry 120, (iv) repeating steps (ii) and (iii) until the last inference operation has been performed, and (v) storing one or more results (e.g., activations) in random access memory 123.

[0080] Figure 4A flow chart of a method according to some example embodiments disclosed is shown. The method includes: at 405, performing an inference operation of a neural network. The steps of performing may include: reading weights from a persistent memory into a random access memory; multiplying the weights by elements of an input feature map to form a first product; and calculating activations based on the first product. The step of reading the weights from the persistent memory into the random access memory may include: reading the weights from the persistent memory into the random access memory via a broadband data connection. The method also includes: at 410, storing the activations in the random access memory. The step of performing an inference operation may include: performing the inference operation in a system including a persistent memory and a control and inference circuit. The control and inference circuit may be connected to the persistent memory via a broadband data connection. The control and inference circuit may include a persistent memory controller and a multiply-accumulate circuit.

[0081] As used herein, "part of" or "a portion of" something means "at least some of" that thing, and as such may mean less than all of that thing or all of that thing. As such, "part of" or "a portion of" a thing includes the entire thing as a special case (i.e., the entire thing is an example of a part of or a portion of a thing). As used herein, when a second quantity is "within Y" of a first quantity X, it means that the second quantity is at least XY, and the second quantity is at most X+Y. As used herein, when a second quantity is "within Y%" of a first quantity, it means that the second quantity is at least (1-Y / 100) times the first quantity, and the second quantity is at most (1+Y / 100) times the first quantity. As used herein, the term "or" should be interpreted as "and / or", so that, for example, "A or B" means either "A" or "B" or "A and B".

[0082] The background provided in the Background section of this disclosure is included only to set context, and the contents of this section are not admitted to be prior art. Any component or any combination of components described (e.g., in any system diagram included herein) may be used to perform one or more operations of any flow chart included herein. In addition, (i) the operations are example operations and may involve various additional steps not explicitly included, and (ii) the chronological order of the operations may vary.

[0083] Each of the terms "processing circuitry" and "means for processing" is used herein to refer to any combination of hardware, firmware, and software employed to process data or digital signals. Processing circuit hardware may include, for example, an application specific integrated circuit (ASIC), a general or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device (such as a field programmable gate array (FPGA)). In a processing circuit, as used herein, each function is performed by hardware that is configured (i.e., hardwired) to perform that function, or by more general hardware (such as a CPU) that is configured to execute instructions stored in a non-transitory storage medium. The processing circuit may be manufactured on a single printed circuit board (PCB) or distributed on several interconnected PCBs. The processing circuit may contain other processing circuits; for example, the processing circuit may include two processing circuits (FPGA and CPU) interconnected on a PCB.

[0084] As used herein, when a method (e.g., adjustment) or a first quantity (e.g., first variable) is said to be "based on" a second quantity (e.g., second variable), it means that the second quantity is an input to the method or affects the first quantity (e.g., the second quantity may be an input (e.g., the only input or one of several inputs) to a function used to calculate the first quantity, or the first quantity may be equal to the second quantity). Or the first quantity may be the same as the second quantity (e.g., stored in the same one or more locations in memory).

[0085] It will be understood that although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, without departing from the spirit and scope of the inventive concept, the first element, first component, first region, first layer or first part discussed herein may be referred to as the second element, second component, second region, second layer or second part.

[0086] For ease of description, spatially relative terms (such as "under", "below", "lower", "lower", "above", "upper", etc.) may be used herein to describe the relationship between an element or feature and another (one or more) elements or (one or more) features as shown in the figure. It will be understood that, in addition to the orientation depicted in the figure, such spatially relative terms are intended to cover different orientations of the device in use or in operation. For example, if the device in the accompanying drawings is turned over, the elements described as being "under" or "under" or "lower" of other elements or features will be positioned as "above" other elements or features. Therefore, the example terms "under" and "lower" may cover the orientations above and below. The device may be positioned in other ways (e.g., rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein should be interpreted accordingly. In addition, it will be understood that when a layer is referred to as being "between" two layers, it may be the only layer between the two layers, or one or more intermediate layers may also be present.

[0087] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the inventive concept. As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation rather than terms of degree, and are intended to account for the inherent deviations in measurements or calculations that one of ordinary skill in the art would recognize.

[0088] As used herein, unless the context clearly indicates otherwise, the singular form is intended to also include the plural form. It will also be understood that the terms "include" and / or "comprise" when used in this specification specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Statements such as "at least one of..." modify the entire column of elements when following a column of elements, rather than modifying the individual elements in the column. In addition, the use of "may" when describing an embodiment of the inventive concept represents "one or more embodiments of the present disclosure". In addition, the term "exemplary" is intended to represent an example or illustration. As used herein, the term "use" may be considered synonymous with the term "utilize".

[0089] It will be understood that when an element or layer is referred to as being “on,” “connected to,” “bound to,” or “adjacent to” another element or layer, it can be directly on, directly connected to, directly bound to, or immediately adjacent to another element or layer, or one or more intervening elements or layers may be present. In contrast, when an element or layer is referred to as being “directly on,” “directly connected to,” “directly bound to,” or “immediately adjacent to” another element or layer, there are no intervening elements or layers.

[0090] Any numerical range described herein is intended to include all sub-ranges of the same numerical precision contained in the range. For example, the range of "1.0 to 10.0" or "between 1.0 and 10.0" is intended to include all sub-ranges between the minimum value 1.0 and the maximum value 10.0 (and including the minimum value 1.0 and the maximum value 10.0) (that is, having a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0 (such as, for example, 2.4 to 7.6)). Similarly, the range described as "within 35% of 10" is intended to include all sub-ranges between the minimum value 6.5 (that is, (1-35 / 100) multiplied by 10) and the maximum value 13.5 (that is, (1+35 / 100) multiplied by 10) (and including the minimum value 6.5 and the maximum value 13.5) (that is, having a minimum value equal to or greater than 6.5 and a maximum value equal to or less than 13.5 (such as, for example, 7.4 to 10.6)). Any maximum numerical limitation recited herein is intended to include all lower numerical limitations subsumed therein, and any minimum numerical limitation stated in this specification is intended to include all higher numerical limitations subsumed therein.

[0091] It will be understood that when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements. As used herein, "generally connected" means connected through an electrical path that may contain any intervening elements (including intervening elements whose presence qualitatively changes the behavior of the circuit). As used herein, "connected" means (i) "directly connected" or (ii) connected with an intervening element, which is an element that qualitatively affects the behavior of the circuit (e.g., a low value resistor or inductor, or a short segment of transmission line).

[0092] Some embodiments may include features of the following numbered statements.

[0093] 1. A system comprising: a first persistent storage; and Control and inference circuits, in: The first persistent memory is connected to the control and inference circuitry via a broadband data connection; and The control and inference circuitry is configured to perform arithmetic operations.

[0094] 2. The system of statement 1, wherein the broadband data connection has a bandwidth of at least 10 gigabytes per second.

[0095] 3. A system according to any of the preceding statements, wherein: The control and inference circuitry includes a first persistent memory controller; The system further comprises: Interface circuits, a second persistent storage, and a second persistent memory controller, The second persistent memory is connected to the second persistent memory controller; The control and inference circuitry is connected to the interface circuitry; and The second persistent memory controller is connected to the interface circuit.

[0096] 4. A system according to any of the preceding statements, further comprising: Universal flash memory storage interface circuit, in: The control and inference circuits are connected to the universal flash storage interface circuit.

[0097] 5. The system of statement 1, further comprising: Universal flash memory storage interface circuit, a second persistent storage, and a second persistent memory controller, in: The control and inference circuitry includes a first persistent memory controller, The control and inference circuits are connected to the universal flash memory storage interface circuit, and The second persistent memory is connected to the universal flash storage interface circuit.

[0098] 6. A system according to any of the preceding statements, further comprising: a second persistent storage, and a second persistent memory controller, in: The control and inference circuitry includes a first persistent memory controller, The first persistent memory includes flash memory, The second persistent memory includes flash memory, The first persistent memory controller includes a first flash memory controller, and The second persistent memory controller includes a second flash memory controller.

[0099] 7. The system of any of the preceding statements, wherein the broadband data connection comprises a Peripheral Component Interconnect Express (PCIe) connection.

[0100] 8. A system according to any of the preceding statements, further comprising: a first interface circuit, and Artificial Intelligence Accelerator, The artificial intelligence accelerator includes an artificial intelligence processing circuit and a second interface circuit. in: The first interface circuit is connected to the second interface circuit, The artificial intelligence accelerator is configured to perform inference operations of a neural network with the assistance of a first persistent memory and control and inference circuits.

[0101] 9. A system according to any of the preceding statements, further comprising: a first interface circuit, and An artificial intelligence accelerator, connected to a first interface circuit, in: The artificial intelligence accelerator is configured to: perform inference operations of the neural network with the assistance of the first persistent memory and the control and inference circuits; and The first persistent memory and the control and inference circuitry are configured to perform an operation selected from the group consisting of pruning, sparsifying, compressing, quantizing, and approximating.

[0102] 10. A system according to any of the preceding statements, wherein: The first persistent memory is part of the first semiconductor die; and The control and inference circuitry is part of the second semiconductor die.

[0103] 11. A system according to any of the preceding statements, wherein: The first persistent memory is part of the first semiconductor die; The control and inference circuitry is part of the second semiconductor die; and The first semiconductor die and the second semiconductor die are part of a stack of dies.

[0104] 12. A system according to any one of the preceding statements, comprising: a random access memory, in: The first persistent memory is part of the first semiconductor die; The control and inference circuitry is part of the second semiconductor die; The random access memory is part of a third semiconductor die; and The first semiconductor die, the second semiconductor die, and the third semiconductor die are part of a stack of dies.

[0105] 13. A system according to any one of the preceding statements, comprising a random access memory, in: The first persistent memory is part of the first semiconductor die; The control and inference circuitry is part of the second semiconductor die; The random access memory is part of a third semiconductor die; and The first semiconductor die is stacked on the second semiconductor die and the third semiconductor die.

[0106] 14. A system according to any one of the preceding statements, comprising a random access memory, in: The control and inference circuits include: a persistent memory controller; and Multiply-accumulate circuit, The first persistent memory is part of the first semiconductor die; The persistent memory controller is part of the second semiconductor die; The multiply-accumulate circuit is part of a third semiconductor die; The random access memory is part of a fourth semiconductor die; and The first semiconductor die is stacked on the second semiconductor die, the third semiconductor die, and the fourth semiconductor die.

[0107] 15. A method comprising: Perform inference operations on the neural network, The steps performed include: Read the weights from persistent storage into random access memory; Multiplying the weights by the elements of the input feature map to form a first product; and The activation is calculated based on the first product, The step of reading the weights from the persistent memory into the random access memory includes: reading the weights from the persistent memory into the random access memory via a broadband data connection.

[0108] 16. The method of statement 15 further comprising storing the activation in a random access memory.

[0109] 17. The method of claim 15 or claim 16, wherein: The step of performing an inference operation includes performing an inference operation in a system comprising: Persistent storage, and Control and inference circuits, The control and inference circuitry is connected to the persistent storage via a broadband data connection; and The control and inference circuits include: a persistent memory controller; and Multiply-accumulate circuit.

[0110] 18. An apparatus comprising: Connectors; a first persistent storage; and Control and inference circuits, in: The first persistent memory is connected to the control and inference circuitry via a broadband data connection; and The connector is suitable for connecting the device to a mobile computing device.

[0111] 19. The apparatus of statement 18, wherein: The control and inference circuitry includes a first persistent memory controller; The device also includes: Interface circuits, a second persistent storage, and a second persistent memory controller, The second persistent memory is connected to the second persistent memory controller; The control and inference circuitry is connected to the interface circuitry; and The second persistent memory controller is connected to the interface circuit.

[0112] 20. The apparatus of claim 18 or claim 19, wherein: The first persistent memory and the control and inference circuitry are configured to perform an operation selected from the group consisting of pruning, sparsifying, compressing, quantizing, and approximating.

[0113] 21. The apparatus of statement 18, wherein: The first persistent memory is part of the first semiconductor die; and The control and inference circuitry is part of the second semiconductor die.

[0114] Although exemplary embodiments of systems and methods for in-memory inference have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it should be understood that systems and methods for in-memory inference constructed according to the principles of the present disclosure may be implemented in ways other than those specifically described herein. The invention is also defined in the above claims and their equivalents.

Claims

1. A system for storing machine learning operations, comprising: a first persistent storage; as well as Control and inference circuits, in: The first persistent memory is connected to the control and inference circuitry via a broadband data connection; and The control and inference circuitry is configured to perform arithmetic operations.

2. The system of claim 1, wherein: The control and inference circuitry includes a first persistent memory controller; The system further comprises: Interface circuits, a second persistent storage, and a second persistent memory controller, The second persistent memory is connected to the second persistent memory controller; The control and inference circuitry is connected to the interface circuitry; and The second persistent memory controller is connected to the interface circuit.

3. The system according to claim 1, further comprising: Interface circuits, in: The interface circuit includes a serial interface; and The control and inference circuits are connected to the interface circuits.

4. The system according to claim 1, further comprising: Interface circuit; a second persistent storage; as well as a second persistent memory controller, in: The control and inference circuitry includes a first persistent memory controller; The interface circuit includes a serial interface; The control and inference circuitry is connected to the interface circuitry; and The second persistent memory is connected to the interface circuit.

5. The system according to claim 1, further comprising: a second persistent storage, and a second persistent memory controller, in: The control and inference circuitry includes a first persistent memory controller, The first persistent memory includes flash memory, The second persistent memory includes flash memory, The first persistent memory controller includes a first flash memory controller, and The second persistent memory controller includes a second flash memory controller.

6. The system according to claim 1, wherein: Broadband data connections include serial connections.

7. The system of claim 1, further comprising: a first interface circuit, and Artificial Intelligence Accelerator, The artificial intelligence accelerator includes an artificial intelligence processing circuit and a second interface circuit. in: The first interface circuit is connected to the second interface circuit, The artificial intelligence accelerator is configured to perform inference operations of a neural network with the assistance of the first persistent memory and the control and inference circuits.

8. The system of claim 1, further comprising: a first interface circuit, and The artificial intelligence accelerator is connected to the first interface circuit, in: The artificial intelligence accelerator is configured to perform inference operations of the neural network with the assistance of the first persistent memory and the control and inference circuitry; and The first persistent memory and the control and inference circuitry are configured to perform an operation selected from the group consisting of pruning, sparsification, compression, quantization, and approximation.

9. The system of claim 1, wherein: The first persistent memory is part of the first semiconductor die; and The control and inference circuitry is part of the second semiconductor die.

10. The system of claim 1, wherein: The first persistent memory is part of the first semiconductor die; The control and inference circuitry is part of the second semiconductor die; and The first semiconductor die and the second semiconductor die are part of a stack of dies.

11. The system of claim 1 further comprising a random access memory, in: The first persistent memory is part of the first semiconductor die; The control and inference circuitry is part of the second semiconductor die; The random access memory is part of a third semiconductor die; and The first semiconductor die, the second semiconductor die, and the third semiconductor die are part of a stack of dies.

12. The system of claim 1 further comprising a random access memory, in: The first persistent memory is part of the first semiconductor die; The control and inference circuitry is part of the second semiconductor die; The random access memory is part of a third semiconductor die; and The first semiconductor die is stacked on the second semiconductor die and the third semiconductor die.

13. The system according to claim 1, comprising: Random Access Memory, in: The control and inference circuits include: a persistent memory controller; and Multiply-accumulate circuit, The first persistent memory is part of the first semiconductor die; The persistent memory controller is part of the second semiconductor die; The multiply-accumulate circuit is part of a third semiconductor die; The random access memory is part of a fourth semiconductor die; and The first semiconductor die is stacked on the second semiconductor die, the third semiconductor die, and the fourth semiconductor die.

14. A method for storing machine learning operations, comprising: Perform inference operations on the neural network, The steps performed include: Read the weights from persistent storage into random access memory; Multiplying the weights by the elements of the input feature map to form a first product; and The activation is calculated based on the first product, The step of reading the weights from the persistent memory into the random access memory includes: reading the weights from the persistent memory into the random access memory via a broadband data connection.

15. The method according to claim 14, further comprising: The activations are stored in random access memory.

16. The method according to claim 14 or 15, wherein: The step of performing an inference operation includes performing an inference operation in a system comprising: Persistent storage, and Control and inference circuits, The control and inference circuitry is connected to the persistent storage via a broadband data connection; and The control and inference circuits include: a persistent memory controller; and Multiply-accumulate circuit.

17. An apparatus for storing machine learning operations, comprising: Connectors; a first persistent storage; as well as Control and inference circuits, in: The first persistent memory is connected to the control and inference circuitry via a broadband data connection; and The connector is suitable for connecting the device to a mobile computing device.

18. The device according to claim 17, wherein: The control and inference circuitry includes a first persistent memory controller; The device also includes: Interface circuit; a second persistent storage; and a second persistent memory controller, The second persistent memory is connected to the second persistent memory controller; The control and inference circuitry is connected to the interface circuitry; and The second persistent memory controller is connected to the interface circuit.

19. The apparatus of claim 17, wherein: The first persistent memory and the control and inference circuitry are configured to perform an operation selected from the group consisting of pruning, sparsifying, compressing, quantizing, and approximating.

20. The device according to any one of claims 17 to 19, wherein: The first persistent memory is part of the first semiconductor die; and The control and inference circuitry is part of the second semiconductor die.

Images