Reasoning in Memory

Abstract

Some aspects of the present disclosure provide a method for processing data using an expansion memory module with computational cores, the method including receiving data in the expansion memory module from a host processing system, storing the data in a host processing system addressable memory, transferring the data from the host processing system addressable memory to a computational core addressable memory, processing the data using the computational cores on the expansion memory module to generate processed data, transferring the processed data from the computational core addressable memory to the host processing system addressable memory, and providing the processed data to the host processing system via the host processing system addressable memory.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. patent application Ser. No. 16 / 883,869, filed May 26, 2020, the entire contents of which are incorporated herein by reference.

[0002] Aspects of the present disclosure relate to performing machine learning and artificial intelligence tasks in non-traditional computing hardware, and in particular performing such tasks in memory hardware. [Background technology]

[0003] The demand for machine learning and artificial intelligence capabilities is growing exponentially in the era of big data. Historically, machine learning has been used to generate models that may then generate inferences for artificial intelligence tasks such as prediction, classification, etc. As the demand for inference capabilities grows, manufacturers of computing hardware are seeking to expand the density of inference capabilities in existing computational platforms such as desktop computers and servers, as well as other emerging types of processing systems such as mobile devices and edge processing devices.

[0004] Traditionally, machine learning and artificial intelligence "accelerators" have been added to systems through hardware expansion interfaces (e.g., PCIe slots on a motherboard) to extend the capabilities of the underlying computational infrastructure. Unfortunately, using such hardware expansion interfaces for accelerators means that those same interfaces cannot be used for other purposes, such as networking, graphics rendering, security, voice processing, and other general computational tasks. Thus, adding additional computational hardware optimized for machine learning and artificial intelligence to an existing processing system often comes at the expense of other critical computational components. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, what is needed is a system and method for adding additional machine learning and artificial intelligence computing power to existing processing systems without sacrificing other critical components. [Means for solving the problem]

[0006] Some aspects of the disclosure provide an expanded memory module including a compute core and one or more dual mode memory elements, the expanded memory module configured to allocate a first subset of memory among the one or more dual mode memory elements as host processing system addressable memory and allocate a second subset of memory among the one or more dual mode memory elements as compute core addressable memory, receive data from the host processing system, process the data with the compute core to generate processed data, and provide the processed data to the host processing system via the first subset of memory.

[0007] A further aspect provides a method for processing data using an expansion memory module with computational cores, the method including initializing the expansion memory module by allocating a first subset of memory of the expansion memory module as host processing system addressable memory and allocating a second subset of memory of the expansion memory module as computational core addressable memory, receiving data in the expansion memory module from the host processing system, processing the data using the computational cores on the expansion memory module to generate processed data, and providing the processed data to the host processing system via the first subset of memory.

[0008] A further aspect provides a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a host processing system, cause the processing system to perform a method for processing data using an expansion memory module comprising computational cores, the method including: initializing the expansion memory module by allocating a first subset of memory of the expansion memory module as host processing system addressable memory and allocating a second subset of memory of the expansion memory module as computational core addressable memory; receiving data at the expansion memory module from the host processing system; processing the data using the computational cores on the expansion memory module to generate processed data; and providing the processed data to the host processing system via the first subset of memory.

[0009] The following description and the related drawings set forth in detail certain illustrative features of the one or more embodiments.

[0010] The accompanying drawings illustrate several aspects of the one or more embodiments and therefore should not be considered as limiting the scope of the disclosure. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an example of an expanded memory module including a memory integrated accelerator. [Diagram 2] FIG. 2 illustrates an exemplary processing system including an expansion memory module, as described with respect to FIG. [Figure 3A] FIG. 2 illustrates an exemplary configuration of an expansion memory module. [Figure 3B] FIG. 2 illustrates an exemplary configuration of an expansion memory module. [Figure 3C] FIG. 2 illustrates an exemplary configuration of an expansion memory module. [Figure 4] FIG. 1 illustrates an exemplary method for transmitting data to a computational core memory element on an expansion memory module. [Diagram 5] FIG. 1 illustrates an exemplary method for transferring data from a computational core memory element on an expansion memory module. [Figure 6] FIG. 1 illustrates an exemplary method for processing data using computational cores on an expansion memory module. [Figure 7] FIG. 1 illustrates an exemplary method for constructing an expansion memory module. [Figure 8] FIG. 2 illustrates an exemplary memory mapping for processing a system including an expansion memory module. [Figure 9] FIG. 1 illustrates an exemplary method for processing data using an expanded memory module with a computational core. [Figure 10] FIG. 1 illustrates an exemplary electronic device that may be configured to perform data processing using an expansion memory module. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] For ease of understanding, the same reference numbers have been used, where possible, to designate identical elements common to the figures. It is contemplated that elements and features of one embodiment may be advantageously incorporated in other embodiments without further recitation.

[0013] Aspects of the present disclosure provide systems and methods for adding additional machine learning and artificial intelligence computing power to memory hardware in processing systems such as computers, servers, and other computer processing devices.

[0014] A conventional computer processing system may include a processor that is connected to a memory module, such as a DRAM module, via an interface, such as a DIMM interface, and to other peripheral devices via other interfaces, such as a PCIe interface. As the demand for machine learning and artificial intelligence "accelerators" has increased, one solution for existing computer processing systems has been to connect such accelerators to available system interfaces, such as available PCIe slots. However, the number of such peripheral interface slots in any given system may be limited; in particular, conventional computer processing systems may generally include a larger number of DIMM slots than PCIe slots.

[0015] Described herein is an enhanced memory module with an integrated accelerator configured to perform machine learning and artificial intelligence tasks such as training and inference. The memory integrated accelerator may be implemented in an industry standard dual in-line memory module (DIMM), such as a Load Reduced DIMM (LRDIMM) or a Registered DIMM (RDIMM), for example, using a standard memory interface for data and instruction communication. In some embodiments, the enhanced memory module may be referred to as, for example, an "ML-DIMM," an "AI-DIMM," or an "Inference DIMM."

[0016] In various embodiments, the memory integrated accelerator may be further configured as a system-on-chip (SoC) with one or more processing cores and integrated into an extended memory module. The processing cores enable on-memory processing of data beyond traditional storage and retrieval. In particular, the processing cores may be configured to perform on-memory machine learning and artificial intelligence tasks.

[0017] Additionally, the expansion memory modules described herein may be configured to partition and allocate memory elements and / or memory space within each expansion memory module between a host processing system, as in a standard memory module configuration, and a memory integrated accelerator for on-memory machine learning and artificial intelligence tasks. In some embodiments, a firmware element may configure the allocation of memory between host processing and on-memory processing at boot time.

[0018] The partitioning of memory within the expansion memory module may also be dynamically configured by programs, applications installed on the device containing the expansion memory module, such that the performance of the entire processing system may be adjusted for various tasks and performance demands, and advantageously, the expansion memory module may be configured to behave as a regular memory module, maximizing the amount of memory available to the host processing system, or may be configured to allocate some or all of its memory to memory integrated accelerators, maximizing the machine learning and artificial intelligence computing capabilities of the host processing system.

[0019] Thus, the embodiments described herein advantageously allow additional computational resources to be utilized by a processing system for computational tasks, such as machine learning and artificial intelligence tasks, without removing or preventing the use of other critical components due to the lack of an available interface (e.g., PCIe), without adding or modifying additional physical interfaces, and without having to implement entirely new interfaces. The additional computational resources greatly increase the capabilities for existing processing systems as well as new processing systems. Moreover, the embodiments described herein advantageously utilize a dual operating mode architecture that allows a user to select different operating modes in different situations to maximize utilization of the processing system.

[0020] The embodiments described herein may be used in many contexts. For example, server-based data centers may implement the embodiments described herein to procure fewer servers while obtaining the same amount of computing capacity, which saves on equipment costs, power and cooling costs, space costs, and also results in a more power-efficient processing center. As another example, desktop and / or consumer computing systems may implement the embodiments described herein to bring higher data processing power to home computers, rather than relying on external data processing power (e.g., as provided by cloud processing services). As yet another example, Internet of Things (IoT) devices may implement the embodiments described herein to extend data processing power to entirely new types of devices, further enabling processing (e.g., machine learning and artificial intelligence inference) at the "edge."

[0021] Exemplary Expansion Memory Module Using a Memory Integrated Processing Core FIG. 1 illustrates an example of an expanded memory module 102 that includes a memory integrated accelerator 104 .

[0022] In some embodiments, the expansion memory module 102 is fabricated according to a standardized form factor, such as the LR-DIMM form factor, or another standardized form factor for memory modules. Utilizing a standardized form factor for the expansion memory module 102 allows it to be integrated into an existing processing system without modifications to the underlying system interface or memory access protocol. In this example, the memory module 102 includes pins 116 for interfacing with a DIMM slot in the processing system.

[0023] The memory module 102 includes multiple memory devices 112A-F, which may be, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), low power double data rate (LPDDR) DRAM, high bandwidth memory (HMB), graphics DDR (GDDR) memory, and others. In some examples, the memory devices 112A-F may include a mix of different types of memory devices.

[0024] Each of the memory devices 112A-F is connected to an adjacent memory buffer 114A-F, which is used to buffer data read from and written to the memory devices 112A-F by a host processing system (not shown).

[0025] The memory devices 112A-F may generally be referred to as dual-mode or dual-purpose memory devices because all or a portion of their memory may be allocated to the memory integrated accelerator 104 for performing on-memory processing and / or to the host processing system for performing conventional system memory functions. Figures 3A-3C illustrate some examples of different memory allocations.

[0026] The memory integrated accelerator 104 includes a flash memory 106 configured to store firmware, which may be configured to initialize the memory module 102. For example, the firmware may be configured to initialize the compute cores 108 and any other logic on the expansion memory module. In addition, a memory controller (not shown) may be configured to initialize the memory devices 112A-F for use by the compute cores 108 and / or the host processing system.

[0027] In some embodiments, the initialization of memory devices 112A-F may be handled by accelerator 104, which may improve (e.g., by reducing) the boot time of the host processing system. In such cases, the memory driver of the host processing system only needs to initialize the memory interface.

[0028] In some embodiments, firmware in flash memory module 106 may configure memory module 102 at boot time to allocate (or map) some or all of the memory capacity (e.g., including memory elements and / or memory ranges, addresses, etc.) to accelerator 104. Firmware in flash memory module 106 may be updated such that the amount of memory allocated to accelerator 104 can be changed between run-times. In embodiments in which memory (e.g., ranges of memory addresses) from memory elements 112A-F are allocated at boot time, the incorporating processing system only "sees" the remaining available memory (except for memory allocated to accelerator 104) and does not attempt to write to the memory allocated to accelerator 104, thereby avoiding memory conflicts.

[0029] In some embodiments, firmware in the flash memory module 106 may configure an initial memory allocation at boot time, which may then be dynamically changed upon command from the host processing system. In such cases, the operating system of the host processing system may be configured to dynamically reallocate memory used by the processing system to avoid conflicts.

[0030] The accelerator 104 includes one or more computational cores 108, which may be configured to perform various types of processing tasks, including machine learning and artificial intelligence tasks such as training and inference. In some embodiments, one or more of the computational cores 108 may be based on an ARM™ architecture, a reduced instruction set computer (RISC) architecture, or a complex instruction set computer (CISC) architecture, to name a few.

[0031] In some embodiments, the compute core 108 replaces and integrates the functionality of a registry clock driver chip or circuit on a conventional memory module.

[0032] The accelerator 104 further includes multiple memory elements 110A-B, which are dedicated to the computational cores 108, or in other words, single-mode or single-purpose memory elements. Because the memory elements 110A-B are dedicated to the computational cores 108, the memory associated with the memory elements 110A-B is not addressable by the host processing system. The dedicated memory elements 110A-B enable the accelerator 104 to always perform computational tasks. In some examples, the memory elements 110A-B may be configured to load model parameters, such as weights, biases, etc., for machine learning models configured to perform machine learning or artificial intelligence tasks, such as training or inference.

[0033] In some embodiments, the accelerator 104 may be implemented as a system-on-chip (SOC) that is integrated with the memory module 102.

[0034] 1 are merely exemplary, and many other configurations are possible. Additionally, while shown as having all elements on one side of memory module 102, in other embodiments memory module 102 may include additional elements on the opposite side, such as additional memory elements, buffers, compute cores, flash memory modules, etc.

[0035] Exemplary Processing System Including Expansion Memory Module FIG. 2 illustrates an example processing system 200 that includes an expansion memory module, such as 102 described with respect to FIG.

[0036] In this example, processing system 200 implements a software stack that includes an operating system (OS) user space 202, an OS kernel space 212, and a CPU 218. The software stack is configured to interface with expansion memory modules 224A and 224B.

[0037] In general, an operating system may be configured to separate virtual memory between an OS user space, such as 202, and an OS kernel space, such as 212, to provide memory protection and hardware protection from malicious or erroneous software behavior. In this example, OS user space 202 is the memory area in which application software, such as ML / AI applications 208 and accelerator runtime applications 210, execute, while kernel space 212 is the memory space reserved for executing privileged operating system kernels, kernel extensions, and device drivers, such as memory drivers 214 and accelerator drivers 216.

[0038] Further in this example, OS user space 202 includes machine learning and / or artificial intelligence applications 208, which may implement one or more models 204 (e.g., machine learning models such as artificial neural network models) configured to process data 206. Models 204 may include, for example, weights, biases, and other parameters. Data 206 may include, for example, video data, image data, audio data, text data, and other types of data on which models 204 may operate to perform various machine learning tasks, such as image recognition and separation, speech recognition and translation, value prediction, and others.

[0039] The ML / AI application 208 is further configured to interface with the accelerator runtime application 210, which is a user space process configured to direct data processing requests to processing accelerators, such as those found in the expanded memory modules 224A-B. Notably, although a single ML / AI application 208 is shown in FIG. 2, this is by way of example only and other types of processing applications may be configured to reside in the OS user space 202 and take advantage of the expanded memory modules 224A-B.

[0040] Accelerator runtime application 210 is configured to interface with accelerator drivers 216 in OS kernel space 212. Generally, a driver in OS kernel space 212 is software configured to control a particular type of device attached to a processing system, such as processing system 200. For example, a driver may provide a software interface to a hardware device, such as memory management unit 220, that allows the operating system and other computer programs to access the functionality of the hardware device without having to know exact details about the hardware device.

[0041] In this example, the accelerator driver 216 is configured to provide access to the expansion memory modules 224A-B by the memory driver 214. In some embodiments, the accelerator driver 216 is configured to implement a protocol used by the expansion memory modules 224A-B. To this end, the accelerator driver 216 may interact with the memory driver 214 to send commands to the expansion memory modules 224A-B. For example, such commands are configured to transfer data from memory allocated to the host processing system to memory allocated to the accelerators (e.g., to the computational cores) of the expansion memory modules 224A-B and begin processing the data, or transfer data from memory allocated to the accelerators to memory allocated to the host processing system.

[0042] In some embodiments, the accelerator driver 216 can implement a protocol using specific memory addresses to write to or read from the memory of the expanded memory modules 224A-B. In some cases, the memory addresses may be physically mapped to the expanded memory modules. The accelerator driver 216 may also have access to memory addresses on the expanded memory modules 224A-B that are not accessible to the OS user space 202, which can be used to copy data to memory allocated to the accelerators of the expanded memory modules 224A-B and then to the compute cores of those expanded memory modules.

[0043] Further, the accelerator 216 is configured to interface with the memory driver 214. The memory driver 214 is configured to provide low-level access to the memory elements of the expansion memory modules 224A-B. For example, the memory driver 214 may be configured to pass commands generated by the accelerator driver 216 directly to the expansion memory modules 224A-B. The memory driver 214 may be part of the memory management of the operating system, but may be modified to enable the expansion memory module features described herein. For example, the memory driver 214 may be configured to block access to the expansion memory modules 224A-B during the period when data is being copied between memory allocated to the accelerator (e.g., to the computational cores) and memory allocated to the host processing system in the expansion memory modules 224A-B.

[0044] The memory driver 214 is configured to interface with a memory management unit (MMU) 220 , which is a hardware component configured to manage memory and cache operations associated with a processor, such as the CPU 218 .

[0045] The memory management unit 220 is configured to interface with a memory controller 222, which is a hardware component configured to manage the flow of data to and from a main memory of a host processing system. In some embodiments, the memory controller 222 is a separate chip or is integrated into another chip, such as an integral part of a microprocessor, such as the CPU 218 (in which case it may be referred to as an integrated memory controller (IMC)). The memory controller 222 may alternatively be referred to as a memory chip controller (MCC) or memory controller unit (MCU).

[0046] In this example, memory controller 222 is configured to control the flow of data to and from expansion memory modules 224A-B. In some embodiments, expansion memory modules 224A-B may be expanded DIMMs, as described above.

[0047] Expansion memory modules 224A-B each include a driver 226A-B, which enables communication with elements of processing system 200. For example, drivers 226A-B may enable communication with other elements of processing system 200 via a DIMM interface with memory controller 222.

[0048] In general, an ML / AI application 208 may request that processing of data 206 via a model 204 be performed by an expansion memory module through an accelerator runtime application 210, which communicates the request to an accelerator driver 216 and then to a memory driver 214, which may be another conventional memory driver. The memory driver 214 may then route processing instructions and data to one or more of the expansion memory modules 224A-B via a memory management unit 220 and a memory controller 222. This is just one example process flow and other process flows are possible.

[0049] In some embodiments, when multiple expansion memory modules (e.g., 224A-B) are being used for the same application (e.g., ML / AI application 208), accelerator runtime application 210 and accelerator driver 216 may cooperate to manage data across the system. For example, accelerator runtime application 210 may request the entire available accelerator allocated memory (and number of expansion memory modules) from accelerator driver 216, and then partition the workload among the expansion memory modules as efficiently as possible if a single expansion memory module cannot bear the entire workload. As another example, with multiple workloads and multiple expansion memory modules, accelerator runtime application 210 may be configured to partition the workload among the available expansion memory modules to maximize performance.

[0050] CPU 218 may generally be a central processing unit with one or more unique processing cores. While in this example CPU 218 is not executing specific processing requests from the ML / AI application routed to expansion memory modules 224A-B, in other examples CPU 218 may cooperate with expansion memory modules 224A-B to perform parallel processing of requests from ML / AI application 208 (and other applications).

[0051] Additionally, although several aspects of processing system 200 have been shown and described in this example, processing system 200 may include other components. For example, processing system 200 may include additional applications, additional hardware elements (e.g., other types of processing units as described with respect to FIG. 10), etc. FIG. 2 intentionally focuses on several elements of processing system 200 that are relevant to the described embodiment.

[0052] Exemplary Operating Modes for Expansion Memory Modules 3A-3C show exemplary memory allocation configurations for expansion memory modules (eg, 102 in FIG. 1 and 224A-B in FIG. 2).

[0053] 3A, the expansion memory module 302 is configured, for example at boot time by firmware in flash memory 306, to use only its dedicated memory elements 310A-B for all processing by the compute cores 308. In this configuration, the expansion memory module 302 is set to provide a maximum capacity for system memory.

[0054] 3B, the expansion memory module 302 is configured to use its dedicated memory elements 310A-B in addition to the memory space provided by memory elements 312A-C for any processing by the compute cores 308, while the memory space provided by memory elements 312D-F is reserved for the host processing system, like conventional system memory. Thus, in this configuration, the expansion memory module 302 is set up to balance the demands of the host processing system memory with the on-module computing power.

[0055] 3C, the expansion memory module 302 is configured to use its dedicated memory elements 310A-B in addition to the memory space provided by memory elements 312A-F for any processing by the compute cores 308. Thus, in this configuration, the expansion memory module 302 is set up to maximize on-module computing power. Notably, in such a configuration, another expansion memory module or another conventional memory module may provide host processing system addressable memory for host processing system operations.

[0056] In particular, although the example illustrates various memory elements as being allocated to compute cores 308 or to host system memory, this representation is for convenience: for example, virtual memory address spaces may be allocated to either on-memory computation or host system processing, regardless of how those address spaces are allocated (e.g., by page tables) to physical memory addresses in physical memory elements.

[0057] 3A-3C are only a few illustrative possibilities. The amount of host processing system memory space provided by expansion memory modules 302 and allocated to compute cores 308 may be any ratio between 0% and 100% of the available host processing system addressable memory elements. However, in some embodiments, the ratio of memory space allocated to compute cores 308 may be limited such that at least a portion of the memory space on each expansion memory module is reserved for the host system.

[0058] Exemplary Method for Transferring Data to Computational Core Memory Devices on an Expansion Memory Module - Patent application FIG. 4 illustrates an exemplary method for transferring data to a computational core memory element (eg, 102 in FIG. 1, 224A-B in FIG. 2, and 302 in FIG. 3A-3C) on an expansion memory module.

[0059] The method 400 begins at step 402 with receiving a data command over an extended memory protocol to transfer the data to a computational core memory element on an extended memory module. In some embodiments, an accelerator driver, such as that discussed above with respect to FIG. 2, may generate the command over the extended memory protocol.

[0060] Note that in this example, the compute core memory elements may be designated compute core memory elements, such as memory elements 310A-B of Figure 3, or may be dual mode memory elements allocated to the compute core, such as memory elements 312A-C of Figure 3. Additionally, while method 400 is described with respect to transferring data to and from memory elements, this is for ease of illustration. The memory allocated to the compute core may be a range of memory addresses, memory space, or any other logical as well as physical division of memory.

[0061] The method 400 then proceeds to step 404 of blocking commands of the host processing system for access to memory devices on the expansion memory module to prevent any conflicts while processing data commands with the computational cores on the expansion memory module.

[0062] In some embodiments, a memory driver such as that discussed above with respect to FIG. 2 may be configured to block commands of the host processing system.

[0063] Method 400 then proceeds to step 406 where the data bus is set to a busy state to block read requests from the host processing system while the data is being moved on the expansion memory module to prevent any interruptions while processing data commands with the computational cores on the expansion memory module.

[0064] Method 400 then proceeds to step 408 of transferring data from memory elements allocated to the host processing system to memory elements allocated to the compute cores on the expansion memory module. As above, the memory elements allocated to the compute cores may include designated compute core memory elements, such as memory elements 310A-B of FIG. 3, or dual mode memory elements allocated to the compute cores, such as memory elements 312A-C of FIG. 3. Here, the data transferred to the compute core memory elements may include data to be processed as well as instructions on how to process the data. For example, the data may include data to be processed (e.g., 206 of FIG. 2), model data (e.g., 204 of FIG. 2), and processing commands.

[0065] The method 400 then proceeds to step 410 where it is determined whether the data transfer is complete. If not, the method returns to step 408 to continue the data transfer.

[0066] If the transfer is complete in step 410, the method 400 proceeds to step 412 where the busy state is cleared from the data bus.

[0067] The method 400 proceeds to step 414, where the host processing system's commands to the memory devices allocated to the host processing system are unblocked so that normal memory operations with the host processing system can continue.

[0068] Method 400 then proceeds to step 416 of processing the data using one or more computational cores of the expansion memory module, such as those described in more detail below with respect to Figure 6. For example, processing the data may include performing machine learning tasks such as training or inference.

[0069] Exemplary Method for Transferring Data from Computational Core Memory Devices on an Expansion Memory Module - Patent application 5 illustrates an example method 500 for transferring data from a compute core onto an expansion memory module (e.g., 102 of FIG. 1, 224A-B of FIG. 2, and 302 of FIG. 3A-3C). For example, method 500 may be performed after processing of the data is completed in step 416 of FIG.

[0070] Method 500 begins at step 502 with receiving a result command via the expanded memory protocol to transfer data from a compute core memory element on the expanded memory module. Note that in this example, the compute core memory element may be a designated compute core memory element, such as memory elements 310A-B of FIG. 3, or may be a dual-mode memory element allocated to a compute core, such as memory elements 312A-C of FIG.

[0071] The method 500 then proceeds to step 504 of blocking commands of the host processing system for access to memory devices on the expansion memory module to prevent any conflicts while processing data commands on the expansion memory module.

[0072] The method 500 then proceeds to step 506, where the data bus is set to a busy state to block read requests from the host processing system while the data is being moved on to the expansion memory module.

[0073] Method 500 then proceeds to step 508 of transferring the data from the compute core memory elements to memory elements allocated to the host processing system on the expansion memory module. As above, the memory elements allocated to the compute core may include designated compute core memory elements, such as memory elements 310A-B of FIG. 3, or dual-mode memory elements allocated to the compute core, such as memory elements 312A-C of FIG.

[0074] The method 500 then proceeds to step 510 where it is determined whether the transfer is complete. If not, the method returns to step 508 to continue the data transfer.

[0075] If the transfer is complete in step 510, the method 500 proceeds to step 512 where the busy state is cleared from the data bus.

[0076] Finally, the method 500 proceeds to step 514, where the host processing system's commands to the memory devices allocated to the host processing system are unblocked so that normal memory operations with the host processing system can continue.

[0077] Exemplary Method for Processing Data with Computational Cores on an Expansion Memory Module - Patent application FIG 6 illustrates an example method 600 for processing data using a computational core on an expansion memory module. For example, processing data using a computational core of an expansion memory module (e.g., 102 of FIG 1, 224A-B of FIG 2, and 302 of FIG 3A-3C) may occur after transferring data to a computational core memory element on the expansion memory module as described above with respect to FIG 4. The data to be processed may be generated by an application, a sensor, another processor, or the like. For example, the data to be processed may comprise image, video, or audio data.

[0078] The method 600 begins at step 602 with receiving a command from a requestor to process data on the expansion memory module. For example, the requestor may be a data processing application, such as the ML / AI application 208 of FIG.

[0079] In some embodiments, the requester queries the accelerator runtime application (e.g., 210 of FIG. 2) to determine how many expansion memory modules are configured in the system, as well as how much memory on those expansion memory modules is allocated to running acceleration-based workloads. The accelerator runtime application may then query the accelerator driver (e.g., 216 of FIG. 2) to obtain this information. This information allows the requester to determine whether the workload can fit into a configured amount of memory and run the workload accordingly. If the amount of available memory is insufficient for the workload, the requester may reconfigure the workload, such as reducing the size of the workload or splitting the workload, to make use of the available memory.

[0080] Method 600 then proceeds to step 604 of loading the data into a computational core memory element in the expansion memory module. As described above, the computational core memory element may be a dedicated memory element or a dynamically allocated memory element. In some embodiments, step 604 is performed according to method 400 described with respect to FIG.

[0081] In some embodiments, a requester (e.g., ML / AI application 208 of FIG. 2) sends data to an accelerator runtime application (e.g., 210 of FIG. 2), which forwards the data to an accelerator driver (e.g., 216 of FIG. 2). The accelerator driver may then send a protocol command to load the data to a memory driver (e.g., 214 of FIG. 2), which then sends the command to an expansion memory module (e.g., 224A-B of FIG. 2).

[0082] The method 600 then proceeds to step 606 of receiving a processing command via an expanded memory protocol, as described in more detail below. The processing command may be configured to cause a processing core of the expanded memory module to perform data processing on data stored in the compute core memory elements.

[0083] In some embodiments, after loading the data, the accelerator driver (e.g., 216 in FIG. 2) sends protocol commands for processing the data to the memory driver (e.g., 214 in FIG. 2), which then forwards the commands to the expansion memory module (e.g., 224A-B in FIG. 2) for execution.

[0084] Method 600 then proceeds to step 608 of processing the data stored in the compute core memory elements by the compute cores on the expanded memory module. In some embodiments, this processing may involve machine learning and / or artificial intelligence tasks, such as training or inference. Furthermore, in some embodiments, machine learning tasks may be processed in parallel across multiple compute cores on a single expanded memory module, as well as across multiple expanded memory modules.

[0085] The method 600 then proceeds to step 610 , where if processing of the workload is not yet complete, the method 600 returns to step 608 .

[0086] If, at step 610, processing of the workload is complete, the method 600 proceeds to step 612 where a result signal is sent via the extended memory protocol.

[0087] In some embodiments, the accelerator driver (e.g., 216 of FIG. 2) may periodically query the expansion memory module (e.g., 224A-B of FIG. 2) to determine when processing is complete. Once processing is complete, the expansion memory module may output a signal on the bus indicating that processing is complete. The accelerator driver then observes the result signal the next time the accelerator driver queries the expansion memory module.

[0088] Method 600 then proceeds to step 612, where the processed data is moved from the compute core memory devices to memory devices allocated to the host processing system on the expansion memory module. For example, with reference to Figure 3B, data may be moved from one or more of memory devices 310A-B and 312A-C allocated to the compute cores to one or more of memory devices 312D-F allocated to the host processing system.

[0089] In some embodiments, after processing is complete, the accelerator driver (e.g., 216 in FIG. 2) sends a protocol command to the memory driver (e.g., 214 in FIG. 2) to retrieve the data. The memory driver then passes this command to the expansion memory module, which then copies the processed data to the host processing system addressable memory.

[0090] The method 600 then terminates at step 614 with the processed data being provided to the requester from memory elements allocated to the host processing system on the expansion memory module.

[0091] In some embodiments, the accelerator driver (e.g., 216 in FIG. 2) sends commands to the memory driver (e.g., 214 in FIG. 2) to read the processed data. The memory driver then forwards these read requests to the expansion memory module. The accelerator driver returns these results to the requester by way of the accelerator runtime application (e.g., 210 in FIG. 2).

[0092] Exemplary Aspects of the Extended Memory Protocol An extended memory protocol may be implemented to interact with an extended memory module (e.g., 102 in FIG. 1 or 224A-B in FIG. 2 or 302 in FIG. 3A-C). In some embodiments, the extended memory protocol may be based on or an extension of the Joint Electron Device Engineering Council (JEDEC) protocol.

[0093] In some embodiments, the extended memory protocol may implement various commands for interfacing with a memory integrated accelerator, such as accelerator 104 described with respect to FIG.

[0094] For example, the extended memory protocol may include an “initialize” signal (or command) configured to load firmware from a flash memory module, such as module 106 described above with respect to FIG. 1, and initialize the memory integrated accelerator.

[0095] The expanded memory protocol may further include a “data” signal (or command) configured to initiate a process for transferring data from memory elements allocated to the system memory (e.g., 312D-F of FIG. 3B) on the expanded memory module (e.g., 102 of FIG. 1 , 224A-B of FIG. 2 , and 302 of FIG. 3A-3C ) to the computational core memory elements (e.g., 312A-C of FIG. 3B ), which may be statically or dynamically allocated as described above, such as in the manner described with respect to FIG. 4 .

[0096] The "data" signal may have variations such as "move all data," in which all data between a start address and an end address is moved between the system allocated memory and the compute core allocated memory, "move x bytes," in which x bytes of data are moved from the system allocated memory to the compute core allocated memory, and "move for x cycles," in which x number of cycles are used to transfer data from the system allocated memory to the compute core allocated memory.

[0097] The expanded memory protocol may further include a "process" signal (or command) configured to cause the computational cores of the expanded memory module (e.g., 102 in FIG. 1, 224A-B in FIG. 2, and 302 in FIG. 3A-3C) to begin executing instructions stored in memory allocated by the computational cores, such as by the method described with respect to FIG. 6.

[0098] The extended memory protocol may further include a “result” signal (or command) configured to initiate a process for transferring data from the compute core-allocated memory (e.g., 312A-C of FIG. 3B) to the system-allocated memory (e.g., 312D-F of FIG. 3B) so that other portions of the host processing system (e.g., system 200 of FIG. 2) can access the data, such as in the manner described with respect to FIG. 5.

[0099] Thus, in one example, the accelerator driver (e.g., 216 in FIG. 2) sends a “data” signal and the memory driver (e.g., 214 in FIG. 2) forwards the signal to the expansion memory module (e.g., 224A-B in FIG. 2). The accelerator driver tells the memory driver to block traffic, which can be a single command using the “data” command or a separate command. The memory driver then blocks all traffic that does not come from the accelerator driver. In some cases, traffic may be held or sent back for retry. The expansion memory module may also set a value in a buffer to output a data pattern indicating memory transfer status. For example, the accelerator driver may request data from the expansion memory module (which the memory driver forwards), and if the data pattern is “busy”, the accelerator driver waits and retries, and if the data pattern is “done”, the accelerator driver tells the memory driver to allow traffic again.

[0100] Exemplary Method for Constructing an Expansion Memory Module FIG. 7 illustrates an exemplary method 700 for configuring an expansion memory module (eg, 102 of FIG. 1, 224A-B of FIG. 2, and 302 of FIG. 3A-3C).

[0101] Method 700 begins at step 702 with placing dual mode memory elements on an expansion memory module, such as memory elements 112A-F on expansion memory module 102 of Figure 1. In this example, "dual mode" refers to each memory element being capable of being used for system memory in a conventional manner as well as for local memory for on-module compute cores, such as 108 of Figure 1. The memory elements do not need to be physically modified to operate in both modes.

[0102] The method 700 then proceeds to step 704 of placing a compute core on the expansion memory module, such as compute core 108 of FIG.

[0103] The method 700 then proceeds to step 706 of connecting command / address lines between the compute core and each dual-mode memory device. The command / address lines may be configured to carry signals that are used to determine what portion of the memory is accessed and what the command is.

[0104] The method 700 then proceeds to step 708 of connecting data lines between each dual-mode device and a computational core (for use in the local processing mode) and a memory buffer (for use in the system memory mode).

[0105] The method 700 then proceeds to step 710 of locating the computational core memory elements on an expansion memory module, such as memory elements 110A-B of expansion memory module 102 of FIG.

[0106] The method 700 then proceeds to step 712 of connecting command / address and data lines between the compute core and the compute core memory devices.

[0107] The method 700 then proceeds to step 714 of placing the flash memory module on an expansion memory module, such as flash memory module 106 of expansion memory module 102 of FIG.

[0108] The method 700 then proceeds to step 716 of connecting the flash memory module to a compute core, such as compute core 108 of expansion memory module 102 of FIG.

[0109] The method 700 then proceeds to step 718 of writing firmware to a flash memory module to initialize an expansion memory module, such as expansion memory module 102 of FIG.

[0110] In particular, method 700 is merely one high-level example of how to construct an expansion memory module, such as expansion memory module 102 of Figure 1, and many other examples exist, and for the sake of clarity, discussion of specific manufacturing techniques is omitted.

[0111] Exemplary memory mapping for a processing system including an expansion memory module FIG. 8 illustrates an exemplary memory mapping for a processing system including an expansion memory module (eg, 102 in FIG. 1, 224A-B in FIG. 2, 302 in FIG. 3A-3C).

[0112] In the illustrated example, the processing system has six memory module slots (e.g., six DIMM slots), with each memory module in each DIMM slot having 8 GB of system addressable memory, for a total of 48 GB of system memory. The memory modules in each DIMM slot may be conventional memory modules or expansion memory modules, so that the system as a whole may be in a "hybrid configuration" having a mix of conventional and expanded memory modules, or in an "expanded configuration" with all expansion memory modules. The expansion memory modules may be configured to allocate memory between the host processing system and on-memory processing elements (e.g., to compute cores such as 108 in FIG. 1), so that the more expansion memory modules there are in the system, the more total memory available to be allocated for on-memory processing.

[0113] Further, in this example, a memory allocation threshold is implemented such that up to half of the memory in any expansion memory module can be allocated to on-memory processes (e.g., to compute cores such as 108 in FIG. 1), with the remaining memory allocated to the host processing system. In other embodiments, the memory allocation threshold may not be implemented or may be implemented with a different value. The memory threshold may generally be established to ensure that the host processing system maintains sufficient allocated memory to operate properly.

[0114] Memory maps 802 or 804 show example memory allocations in a processing system having a hybrid configuration of memory modules. Memory maps 806 and 808 show example memory allocations in a processing system having an expanded configuration, where all memory modules are expanded memory modules. In the illustrated example, the memory allocations in the expanded configurations of 806 and 808 include more memory allocated to the compute cores for on-memory processing.

[0115] Further, the memory allocations in memory maps 806 and 808 are specific to the compute cores of each expansion memory module (e.g., expansion memory modules 0-5). However, in other examples, memory may be allocated generally to the compute cores as a set, such as in the exemplary memory maps 802 and 804, rather than specifically as in the exemplary memory maps 806 and 808.

[0116] In general, having more expansion memory modules increases the amount of memory that can be allocated to on-memory processing, thereby improving the host processing system's ability to perform AI and ML specific tasks such as training and inference, as well as increasing the flexibility in which the overall system memory can be configured.

[0117] In the event of dynamic reallocation of memory between the host processing system and the on-memory processes (e.g., via an accelerator such as 104 in FIG. 1), the memory map may be updated accordingly. For example, memory map 802 may be updated to memory map 806, or other combinations.

[0118] The examples shown in FIG. 8 are intended to illustrate only a few examples of various memory configurations using expansion memory modules, and other memory configurations are possible.

[0119] Exemplary Method for Processing Data Using Expansion Memory Modules - Patent application FIG. 9 illustrates an example method 900 for processing data using an expansion memory module (e.g., 102 in FIG. 1, 224A-B in FIG. 2, and 302 in FIG. 3A-3C) with a computation core (e.g., 108 in FIG. 1). In some embodiments, the computation core may be part of a memory integrated accelerator, such as 104 in FIG. 1. In some embodiments, the computation core may comprise multiple processing elements, such as multiple individual processing cores. In some embodiments, the expansion memory module comprises a load reduced dual in-line memory module (LRDIMM).

[0120] The method 900 begins at step 902 with initializing the expansion memory module by allocating a first subset of the expansion memory module as host processing system addressable memory and allocating a second subset of the expansion memory module as compute core addressable memory, e.g., as described with respect to Figures 3A-3C.

[0121] In some embodiments of the method 900, initialization of the expansion memory module may be performed at boot time, while in other embodiments, initialization of the expansion memory module may be performed dynamically during operation of the host processing system.

[0122] In some embodiments, allocating memory may comprise allocating memory of one or more dual-mode memory devices between the compute cores and the host processing system, such as memory devices 112A-F of Figure 1. For example, the allocation may comprise one or more memory ranges or memory address spaces, such as a virtual address space or a physical address space.

[0123] The method 900 then proceeds to step 904 of receiving data at the expansion memory module from the host processing system. The data may be provided by the host processing system to the expansion memory module for processing on the expansion memory module, such as by a computing core.

[0124] In some embodiments, the received data may include data for processing by the machine learning model and machine learning model parameters, hi some embodiments, the received data may further include processing commands or instructions.

[0125] The method 900 then proceeds to step 906 of storing the received data in a subset of the first memory (host processing system addressable memory). In some embodiments, the host processing system addressable memory comprises memory space allocated to the host processing system in one or more dual mode memory elements on an expansion memory module, such as memory elements 112A-F of FIG.

[0126] The method 900 then proceeds to step 908 of transferring the received data from the first subset of memory (the host processing system addressable memory) to the second subset of memory (the compute core addressable memory).

[0127] In some embodiments, the compute core addressable memory comprises non-host processing system addressable memory, such as memory elements 110A-B of Figure 1. In some embodiments, the compute core addressable memory comprises only non-host processing system addressable memory (such as in the example of Figure 3A).

[0128] The method 900 then proceeds to step 910, where the received data is processed with the computational cores on the expansion memory module to generate processed data.

[0129] The method 900 then proceeds to step 912 of transferring the processed data from the second subset of memory (the compute core addressable system) to the first subset of memory (the host processing system addressable memory).

[0130] The method 900 then proceeds to step 914 of providing the processed data to a host processing system via a subset of the first memory (host processing system addressable memory), such as that described above with respect to FIG.

[0131] In some embodiments, initializing the expansion memory module comprises processing firmware instructions stored in a flash memory module on the expansion memory module, as described above with respect to FIG.

[0132] In some embodiments, as described above with respect to FIG. 4, method 900 further includes setting a data bus of the host processing system to a busy state prior to transferring the data from the first subset of memory (the host processing system addressable memory) to the second subset of memory (the compute core addressable memory) to indicate that the data movement is occurring and to block read requests to the host processing system addressable memory.

[0133] In some embodiments, as described above with respect to FIG. 4, method 900 further includes setting a data bus of the host processing system to an available state to enable read or write requests from the host processing system to the host processing system addressable memory after transferring the data from the first subset of memory (the host processing system addressable memory) to the second subset of memory (the compute core addressable memory).

[0134] In some embodiments, as described above with respect to FIG. 4, the method 900 further includes enabling blocking of host processing system memory commands for the expansion memory module before transferring data from the first subset of memory (the host processing system addressable memory) to the second subset of memory (the compute core addressable memory).

[0135] In some embodiments, as described above with respect to FIG. 4, the method 900 further includes disabling blocking of host processing system memory commands for the expansion memory module after transferring the data from the first subset of memory (the host processing system addressable memory) to the second subset of memory (the compute core addressable memory).

[0136] In some embodiments, the method 900 further includes reallocating memory of the expansion memory module by deallocating the first subset of memory and the second subset of memory, allocating a third subset of memory among the one or more dual mode memory elements as host processing system addressable memory, and allocating a fourth subset of memory among the one or more dual mode memory elements as compute core addressable memory, the third subset of memory being different from the first subset and the fourth subset of memory being different from the second subset. In some embodiments, the reallocation is performed after boot time, such as when the host processing system is running. Thus, the reallocation may be referred to as a dynamic or "hot" reallocation of memory.

[0137] For example, in some embodiments, a first allocation, such as that of FIG. 3A, may be dynamically reallocated to a second allocation, such as that of FIG. 3B or FIG. 3C, or others not shown. In some embodiments, the host processing system may be prevented from sending memory commands to the expansion memory module during the reallocation process. In some embodiments, a page table or memory map may be updated after the reallocation so that physical memory that is no longer accessible to the host processing system or physical memory that is newly accessible to the host processing system is appropriately configured. For example, the example memory map of FIG. 8 may be updated after reallocation of memory on the expansion memory module.

[0138] Exemplary Electronic Device for Data Processing Using Expansion Memory Modules Figure 10 illustrates an example electronic device 1000 that may be configured to perform data processing using an expansion memory module, for example, as described herein with respect to Figures 4-6 and 9. In some embodiments, the electronic device 1000 may comprise a server computer.

[0139] The electronic device 1000 includes a central processing unit (CPU) 1002, which may be a multi-core CPU in some examples. Instructions executed on the CPU 1002 may be loaded, for example, from a program memory associated with the CPU 1002 or may be loaded from a memory partition 1024.

[0140] The electronic device 1000 also includes additional processing components adapted to specific functions, such as a graphics processing unit (GPU) 1004, a digital signal processor (DSP) 1006, and a neural processing unit (NPU) 1008.

[0141] An NPU, such as 1008, is generally a specialized circuit configured to implement all the necessary control and computational logic to execute machine learning algorithms, such as algorithms for processing artificial neural networks (ANN), deep neural networks (DNN), random forests (RF), etc. An NPU may alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), a vision processing unit (VPU), or a graph processing unit.

[0142] An NPU may be optimized for training or inference, or in some cases may be configured to balance performance between both. In an NPU capable of performing both training and inference, the two tasks can generally be performed independently.

[0143] For example, an NPU such as 1008 may be configured to accelerate the execution of common machine learning inference tasks such as image classification, machine transformation, object detection, and various other predictive models. In some examples, multiple NPUs may be instantiated on a single chip, such as a system-on-chip (SoC), while in other examples, they may be part of a dedicated neural network accelerator. NPUs designed to accelerate inference are generally configured to operate on complete models. Thus, such NPUs may be configured to input new data and rapidly process the data through already trained models to generate model outputs (e.g., inferences).

[0144] As another example, an NPU may be configured to accelerate common machine learning training tasks, such as processing a test dataset (often labeled or tagged), iterating over the dataset, and adjusting model parameters such as weights and biases to improve model performance. In general, optimization based on erroneous predictions involves backpropagating through layers of a model and determining gradients to reduce prediction error.

[0145] In one implementation, the NPU 1008 is part of one or more of the CPU 1002, GPU 1004, and / or DSP 1006.

[0146] The electronic device 1000 may also include one or more input and / or output devices 1022, such as a screen, a network interface, physical buttons, etc.

[0147] The electronic device 1000 also includes memory 1024, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 1024 includes computer-executable components, which may be executed by one or more of the aforementioned processors of the electronic device 1000.

[0148] The memory 1024 may represent one or more memory modules, such as the expansion memory modules described herein (with respect to FIGS. 1, 2, and 3). For example, the memory 1024 may comprise one or more compute cores 1024I and a flash memory component 1024J. In some cases, the compute core 1024I may comprise an NPU or other type of processor as described herein. Additionally, although not shown, some of the memory 1024 may be accessible only by the compute core 1024J, while some memory may be configurable to be accessible by the processing system 1000 or by the compute core 1024J.

[0149] In this example, memory 1024 includes a receiving component 1024A, a storage component 1024B, a forwarding component 1024C, a processing component 1024D, a sending (or providing) component 1024E, an initialization component 1024F, an allocation component 1024G, and a partitioning component 1024H. The illustrated components and other components not illustrated may be configured to perform various aspects of the methods described herein.

[0150] Although not shown in FIG. 10, the electronic device 1000 may further include one or more data buses for transferring data between various aspects of the electronic device 1000.

[0151] Example clauses Example implementations are described in the following numbered clauses.

[0152] Clause 1: An expansion memory module comprising: a computational core; and one or more dual mode memory elements, the expansion memory module being configured to allocate a first subset of memory among the one or more dual mode memory elements as host processing system addressable memory and allocate a second subset of memory among the one or more dual mode memory elements as computational core addressable memory, receive data from the host processing system, process the data using the computational core to generate processed data, and provide the processed data to the host processing system via the first subset of memory.

[0153] Clause 2: The expansion memory module of clause 1, wherein the expansion memory module is further configured to store received data in a first subset of memory, transfer the received data from the first subset of memory to the second subset of memory before processing the data with the computational cores, and transfer the processed data from the second subset of memory to the first subset of memory after processing the data with the computational cores.

[0154] Clause 3: The expansion memory module of any one of clauses 1 to 2, wherein the second memory subset further comprises one or more single mode memories configured for use by the computational cores.

[0155] Clause 4: The expansion memory module of any one of clauses 1 to 3, further comprising a flash memory module comprising firmware instructions for allocating the first subset of memory and the second subset of memory.

[0156] Clause 5: The expansion memory module of any one of clauses 1 to 4, wherein the expansion memory module is further configured to set a data bus of the host processing system to a busy state before transferring data from the first subset of memory to the second subset of memory to indicate that a transfer of data is occurring.

[0157] Clause 6: The expansion memory module of clause 5, wherein the expansion memory module is further configured to set a data bus of the host processing system to an available state after transferring data from the first subset of memory to the second subset of memory.

[0158] Clause 7: The expansion memory module of any one of clauses 1 to 6, wherein the expansion memory module is further configured to enable blocking of host processing system memory commands for the expansion memory module before transferring data from the first subset of memory to the second subset of memory.

[0159] Clause 8: The expansion memory module of clause 7, wherein the expansion memory module is further configured to disable blocking of host processing system memory commands for the expansion memory module after transferring data from the first subset of memory to the second subset of memory.

[0160] Clause 9: The expansion memory module of any one of clauses 1 to 8, wherein the data comprises data for processing by a machine learning model and machine learning model parameters.

[0161] Clause 10: The expansion memory module of any one of clauses 1 to 9, wherein the expansion memory module comprises a dual in-line memory module (DIMM).

[0162] Clause 11: The expansion memory module of any one of clauses 1 to 10, wherein the expansion memory module is further configured to deallocate the first subset of memory and the second subset of memory, allocate a third subset of memory among the one or more dual mode memory elements as host processing system addressable memory, and allocate a fourth subset of memory among the one or more dual mode memory elements as compute core addressable memory, wherein the third subset of memory is different from the first subset and the fourth subset of memory is different from the second subset.

[0163] Clause 12: A method for processing data using an expansion memory module having a computational core, the method comprising: initializing the expansion memory module by allocating a first subset of the memory of the expansion memory module as host processing system addressable memory and allocating a second subset of the memory of the expansion memory module as computational core addressable memory; receiving data in the expansion memory module from the host processing system; processing the data using the computational core on the expansion memory module to generate processed data; and providing the processed data to the host processing system via the first subset of memory.

[0164] Clause 13: The method of clause 12, further comprising the steps of storing the received data in a first subset of memory, transferring the received data from the first subset of memory to a second subset of memory before processing the data using the computational cores, and transferring the processed data from the second subset of memory to the first subset of memory after processing the data using the computational cores.

[0165] Clause 14: The method of any one of clauses 12 to 13, wherein the first subset of memories and the second subset of memories are associated with one or more dual mode memory elements.

[0166] Clause 15: The method of any one of clauses 12 to 14, wherein initializing the expansion memory module comprises processing firmware instructions stored in a flash memory module on the expansion memory module.

[0167] Clause 16: The method of any one of clauses 12 to 15, wherein the second memory subset is further associated with one or more single-mode memory elements configured for use by the computational core.

[0168] Clause 17: The method of clause 13, further comprising the step of setting a data bus of the host processing system to a busy state prior to transferring the data from the first subset of memory to the second subset of memory to indicate that a transfer of data is occurring.

[0169] Clause 18: The method of clause 17, further comprising setting a data bus of the host processing system to an available state after transferring the data from the first subset of memory to the second subset of memory.

[0170] Clause 19: The method of clause 13, further comprising enabling blocking of host processing system memory commands for the expansion memory module prior to transferring data from the first subset of memory to the second subset of memory.

[0171] Clause 20: The method of clause 19, further comprising disabling blocking of host processing system memory commands for the expansion memory module after transferring the data from the first subset of memory to the second subset of memory.

[0172] Clause 21: The method of any one of clauses 12 to 20, wherein the data comprises data for processing by a machine learning model and machine learning model parameters.

[0173] Clause 22: The method of any one of clauses 12 to 21, wherein the expansion memory module comprises a dual in-line memory module (DIMM).

[0174] Clause 23: The method of any one of clauses 12 to 22, further comprising the steps of deallocating the first subset of memory and the second subset of memory, and allocating a third subset of memory among the one or more dual mode memory elements as host processing system addressable memory and allocating a fourth subset of memory among the one or more dual mode memory elements as compute core addressable memory, wherein the third subset of memory is different from the first subset and the fourth subset of memory is different from the second subset.

[0175] Clause 24: A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a host processing system, cause the host processing system to perform a method for processing data using an expansion memory module having a computational core, the method comprising: initializing the expansion memory module by allocating a first subset of the memory of the expansion memory module as host processing system addressable memory and allocating a second subset of the memory of the expansion memory module as computational core addressable memory; receiving data at the expansion memory module from the host processing system; processing the data using the computational core on the expansion memory module to generate processed data; and providing the processed data to the host processing system via the first subset of memory.

[0176] Clause 25: The non-transitory computer-readable medium of clause 24, the method further comprising the steps of storing the received data in the first subset of memory, transferring the received data from the first subset of memory to the second subset of memory before processing the data with the computational cores, and transferring the processed data from the second subset of memory to the first subset of memory after processing the data with the computational cores.

[0177] Clause 26: The non-transitory computer-readable medium of any one of clauses 24 to 25, wherein the first subset of memory and the second subset of memory are associated with one or more dual-mode memory elements.

[0178] Clause 27: The non-transitory computer-readable medium of any one of clauses 24 to 26, wherein the second memory subset is further associated with one or more single-mode memory elements configured for use by the computational core.

[0179] Clause 28: The non-transitory computer-readable medium of clause 25, the method further comprising setting a data bus of the host processing system to a busy state to indicate that a transfer of data is occurring before transferring the data from the first subset of memory to the second subset of memory.

[0180] Clause 29: The non-transitory computer-readable medium of clause 29, the method further comprising setting a data bus of the host processing system to an available state to permit data transfer from the first subset of memory to the second subset of memory.

[0181] Clause 30: The non-transitory computer-readable medium of clause 25, the method further comprising: enabling blocking of host processing system memory commands for the expansion memory module before transferring data from the first subset of memory to the second subset of memory; and disabling blocking of host processing system memory commands for the expansion memory module after transferring data from the first subset of memory to the second subset of memory.

[0182] Additional Considerations The foregoing description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples described herein are not intended to limit the scope, applicability, or embodiments described in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments. For example, changes may be made in the function and arrangement of the elements described without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components, as appropriate. For example, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of aspects described herein. In addition, the scope of the disclosure is intended to encompass such apparatus or methods practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0183] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0184] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc, as well as any combination having multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other order of a, b, and c).

[0185] The term "determining" as used herein encompasses a wide variety of acts. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. "Determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" may also include resolving, selecting, choosing, establishing, and the like.

[0186] The methods disclosed herein comprise one or more steps or acts for achieving the method. The steps and / or acts of the method may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or acts is specified, the order and / or use of specific steps and / or acts may be modified without departing from the scope of the claims. Furthermore, various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors. In general, where there are operations illustrated in the figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

[0187] The following claims are not intended to be limited to the embodiments set forth herein, but are to be accorded the full scope consistent with the language of the claims. In the claims, reference to an element in the singular shall mean "one or more," and not "one and only," unless expressly stated otherwise. The term "several" refers to one or more, unless expressly stated otherwise. No element of a claim shall be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase "means for," or, in the case of a method claim, unless the element is recited using the phrase "step for." All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly recited in the claims. [Explanation of symbols]

[0188] 102 Expansion Memory Module 106 Flash Memory 108 Computational Cores 110 Memory element 112 Memory elements 114 Memory Buffer 116 pin 200 Processing System 202 OS user space 204 Model 206 Data 208 ML / AI Applications 210 Accelerator Runtime Applications 212 OS kernel space 214 Memory Driver 216 Accelerator Driver 218 CPU 220 Memory Management Unit 222 Memory Controller 224 Expansion Memory Module 226 Driver 302 Expansion Memory Module 306 Flash Memory 308 Computational Core 310 Memory elements 312 Memory elements 802 Memory Map 804 Memory Map 806 Memory Map 808 Memory Map 1000 Electronic Devices 1002 Central Processing Unit (CPU) 1004 Graphics Processing Unit (GPU) 1006 Digital Signal Processor (DSP) 1008 Neural Processing Unit (NPU) 1022 Input / Output Devices 1024 memory 1024A Receiver Components 1024B Storage Components 1024C Transport Component 1024D Processing Components 1024E Sending (or Providing) Components 1024F Initialization Component 1024G Allocation Component 1024H Class Component 1024I Calculation Component 1024J Flash Memory Components

Claims

1. 1. An expansion memory module, comprising: A computing core; one or more dual mode memory elements; Equipped with The expansion memory module is allocating a first subset of memory in the one or more dual mode memory devices as host processing system addressable memory and a second subset of memory in the one or more dual mode memory devices as compute core addressable memory; Receive data from a host processing system; processing the data with the computational cores to generate processed data; providing the processed data to the host processing system via the first subset of memory; 2. An expansion memory module configured to:

2. The expansion memory module is storing the received data in a subset of the first memory; transferring the received data from the first subset of memory to the second subset of memory before processing the data with the computational cores; 2. The expansion memory module of claim 1, further configured to transfer the processed data from the second subset of memory to the first subset of memory after processing the data with the computational cores.

3. 10. The expansion memory module of claim 1, wherein the second subset of memories further comprises one or more single mode memories configured for use by the compute cores.

4. 10. The expansion memory module of claim 1, further comprising a flash memory module comprising firmware instructions for allocating the first subset of memory and the second subset of memory.

5. 2. The expansion memory module of claim 1, further configured to set a data bus of the host processing system to a busy state to indicate that a data transfer is occurring prior to transferring the data from the first subset of memory to the second subset of memory.

6. 6. The expansion memory module of claim 5, further configured to set the data bus of the host processing system to an available state after the expansion memory module transfers the data from the first subset of memory to the second subset of memory.

7. 2. The expansion memory module of claim 1, further configured to enable blocking of host processing system memory commands for the expansion memory module before transferring the data from the first subset of memory to the second subset of memory.

8. 8. The expansion memory module of claim 7, further configured to disable the block of host processing system memory commands for the expansion memory module after the expansion memory module transfers the data from the first subset of memory to the second subset of memory.

9. The data is Data for processing by the machine learning model; and Machine learning model parameters and or the expansion memory module comprises a dual in-line memory module (DIMM); or The expansion memory module is deallocating the first subset of memory and the second subset of memory; allocating a third subset of the memory of the one or more dual mode memory devices as host processing system addressable memory and a fourth subset of the memory of the one or more dual mode memory devices as compute core addressable memory. The method further comprises: the third subset of memory is different from the first subset; the fourth subset of memory is different from the second subset; 2. The expansion memory module of claim 1.

10. 1. A method for processing data using an expansion memory module with a computational core, comprising: initializing the expansion memory module by allocating a first subset of memory of the expansion memory module as host processing system addressable memory and allocating a second subset of memory of the expansion memory module as compute core addressable memory; receiving data in said expansion memory module from a host processing system; processing the data with the computational cores on the expansion memory module to generate processed data; providing the processed data to the host processing system via the first subset of memory; A method comprising:

11. storing the received data in a subset of the first memory; transferring the received data from the first subset of memory to the second subset of memory before processing the data with the computational cores; transferring the processed data from the second subset of memory to the first subset of memory after processing the data with the computational cores; The method of claim 10, further comprising:

12. The method according to claim 1, further comprising the step of setting a data bus of the host processing system to a busy state to indicate that a data transfer is taking place prior to transferring the data from the first subset of memory to the second subset of memory, preferably and setting the data bus of the host processing system to an available state after transferring the data from the first subset of memory to the second subset of memory. The method of claim 11.

13. The method of claim 12, further comprising the step of enabling blocking of host processing system memory commands for said expansion memory module prior to transferring said data from said first subset of memory to said second subset of memory. disabling the block of host processing system memory commands for the expansion memory module after transferring the data from the first subset of memory to the second subset of memory. The method of claim 11.

14. the first subset of memories and the second subset of memories are associated with one or more dual mode memory elements; or initializing the expansion memory module comprises processing firmware instructions stored in a flash memory module on the expansion memory module; or the second subset of memories is further associated with one or more single-mode memory elements configured for use by the computational cores; or The data is Data for processing by the machine learning model; and Machine learning model parameters and or the expansion memory module comprises a dual in-line memory module (DIMM); or deallocating the first subset of memory and the second subset of memory; allocating a third subset of memory in the one or more dual mode memory devices as host processing system addressable memory and allocating a fourth subset of memory in the one or more dual mode memory devices as compute core addressable memory; the third subset of memory is different from the first subset; the fourth subset of memory is different from the second subset; Further comprising the steps: The method of claim 12.

15. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a host processing system, cause the host processing system to perform a method according to any one of claims 10 to 14 for processing data using an expansion memory module with a computational core.

Images