Memory management system and a method

Abstract

A memory management system (1) configured for use with at least one of: one or more neural networks and / or language models is provided. The memory management system comprises a plurality of memory banks (2). The memory management system comprises one more control units (3). The memory management system comprises one or more multiplexers and / or demultiplexers (4). The memory management system comprises one or more processing units (5). The system (1) is configured to dynamically manage power and data flow among the memory banks (2) based on operational needs of the one or more neural networks and / or the one or more language models. The system (1) comprises switching means (6) for switching between active and inactive states of the memory banks (2).

Description

[0001] MEMORY MANAGEMENT SYSTEM AND A METHOD

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to a memory management system and a method for managing multiple memory banks in one or more computing devices. In particular, the memory management system may be arranged for use by the one or more computing devices.

[0004] BACKGROUND

[0005] In today's neural network landscape, large memory capacity is crucial when training or interacting with Language Models such as Large Language Models (LLMs) and Small Language Models (SMLs) and other neural networks. Interacting as used herein may for example mean inference. Typically, data must be stored in memory for Central Processing Unit (CPU), Graphic Processing Unit (GPU), Neural Processing Unit (NPU), and transformer Application-Specific Integrated Circuity (ASIC) access. New language model configurations, such as "mixed experts" and "neural layers," have emerged. Neural layers as used herein is one or more layers of a neural network. Experts as used herein may be one or more parts of a neural network or language model such as a number of specific neural layers, e.g., which may be known to have certain trained features. Mixed experts as used herein may be a router that selects one or more expert models within the neural network, while neural layers involve multiple interconnected layers of processing units. Hence, there is a need to switch between different complete language models quickly. However, if not all experts, layers, or complete models are stored in memory, they need to be loaded for each token processed, resulting in slow performance. To mitigate this issue, it's essential to keep all experts, layers, or complete models in memory. However, the cost of doing so can be prohibitively expensive for small and medium-sized enterprises (SMEs), making it difficult for them to use language model data. Another drawback is the obvious excessive use of memory, thereby limiting any implementation or use of language models such as LLMs or SMLs. Furthermore, the advent of High Bandwidth Memory (HBM) has introduced new possibilities and challenges in memory management for Artificial Intelligence (Al) and machine learning (ML) applications. The development of Compute Express Link (CXL) memory or other high performance memory technologies has provided an additional layer of flexibility and scalability in memory management. CXL allows for a high-bandwidth, low-latency connection between processing units such as CPUs, GPUs, NPUs, and transformer ASICs and disaggregated memory resources. CXL technology enables memory to be shared across multiple processing units, significantly improving memory utilization, reducing latency, and enhancing the overall efficiency of loading and managing large models, layers, and experts. The integration of CXL memory addresses the challenges of scaling memory resources in high-performance computing environments, offering a more cost-effective solution for handling increasingly complex language models and neural networks. While high performance memory technology is emerging as one solution for improving handling neural networks and language models, limitations of how much memory can be used, energy efficiency, cost of memory, and performance of such memory is still an ongoing issue.

[0006] SUMMARY

[0007] As part of developing embodiments herein, it has been identified that processing units handling neural networks such as language models cannot have access to all of memory at a time. This is since memory may be scarce, expensive, and / or it may not be possible technically to connect sufficient memory to one processing unit, in particular for LLMs with different features, i.e. , experts, in different neural layers, where each layer may require a large amount of memory resources.

[0008] An object herein is to improve memory technology.

[0009] According to a first aspect of the disclosure, a memory management system configured for use with at least one of: one or more neural networks and / or language models is provided. The memory management system comprises a plurality of memory banks, one more control units, one or more multiplexers and / or demultiplexers, one or more processing units, and switching means for switching between active and inactive states of the memory banks. The system is configured to dynamically manage power and data flow among the memory banks based on operational needs of the one or more neural networks and / or the one or more language models.

[0010] According to a second aspect of the disclosure, a method for managing multiple memory banks in one or more computing devices is provided. The method may be for use with one or more language models and / or one or more neural networks. The method comprises dividing memory into one or more memory banks, each bank having a size equivalent to one or more expert models, neural layers, or complete language models. The method comprises designating separate memory banks of the one or more memory banks, for each expert, group of experts, neural layer, or complete language model of the one or more language models and / or the one or more neural networks. This means that each memory bank may be responsible for one or more certain features of the one or more neural networks and / or one or more language models. The method comprises, when a decoder / router associated with a language model and / or a neural network selects an expert, group of experts, neural layer, or complete language model, enabling corresponding memory banks of the one or more memory banks for token processing by allocating memory resources of the corresponding memory banks and by disconnecting unused memory banks. Token processing as used herein may be any suitable processing by neural networks or language models. The method further comprises controlling the memory allocation through a control unit. The method may be performed by, with, or on the system according to the first aspect. The method may manage the memory efficiently such that the different memory resources can be enabled, i.e., activated, and disconnected, i.e., deactivated, by switching to the memory bank needed for use by a certain neural network or language model. This is particularly efficient for LLMs which may need many different expert groups of neural layers for handling different aspects of token processing.

[0011] According to the first and / or second aspect, by switching between the inactive and active states, and by dynamically manage power and data flow among the memory banks based on operational needs, the system enables the use of all memory of the memory banks where otherwise the processing units may have been limited to fewer banks. This is since some of them can be kept active and some inactive. Furthermore, by considering the operational needs, the switching means can be used to switch to memory banks needed by the neural networks or memory models when needed, and the others can be powered down, preferably kept in an idle mode such that no memory is lost. Thereby, the memory banks can be switched on and off with different parts of the language models or neural networks when needed. This allows for processing units to have memory close at hand, without having to move in and out memory from storage. When the memory, e.g., of an LLM, can be divided into the different memory banks, such as by designating certain features of experts or neural layers into some of the banks, and enabling the usage of the memory and banks by allocating the memory resources and disconnecting the unused banks by the control unit, e.g., which may perform the action by the switching means, the memory can be used more efficiently.

[0012] The memory management system of embodiments herein handles memory in banks, e.g., any of Video Random Access Memory (VRAM), Dynamic random-access memory (DRAM), High Bandwidth Memory (HBM), and Compute Express Link (CXL) memory. The system that can be integrated with any processing unit e.g., existing CPU, GPU, NPU, or transformer ASIC architectures or included in new designs. The system may be part of or coupled with any computing device. The system may comprise any computing device. The system enables selective activation of memory banks, e.g., VRAM, HBM, and CXL memory modules, to minimize power consumption and maximize computational efficiency. CXL memory module as used herein may mean any suitable memory connected via an interconnect with at least a minimum set speed such as 1 or 2 gigatransfers per second. HBM as used herein may mean any suitable memory with at least a set bandwidth capability such as 1 gigabit per second. HBM may typically comprise three dimensional stacked memory such as stacked synchronous dynamic random-access memory. CXL memory, with its high- bandwidth, low-latency properties, allows for the dynamic sharing of memory resources across multiple processing units, providing a scalable and flexible solution for managing large language models (LLMs) and small language models (SMLs). This integration ensures that layers, experts, or complete models can be loaded, processed, and unloaded efficiently, reducing data transfer times and improving system responsiveness. Both LLMs and SMLs may be neural networks at least partly trained on general language interpretation and / or handling. The LLMs may use more neural layers than the SMLs.

[0013] The system includes integrated solid-state drives (SSDs) for data backup, ensuring data integrity and quick recovery capabilities. Memory banks as used herein, e.g., including those utilizing CXL memory, may loaded or updated from storage media with a neural layer or a complete language model, LLM or SML, as needed. Additionally, LLM models, layers, and memory can be stored temporarily on storage media, e.g., connected to a PCIe slot or CXL memory of the system of the first aspect. The PCIe slot and / or CXL may allow for faster transfer between the memory banks and the storage media. The system of the first aspect may support both single token and multiple token processing. The control unit may dynamically allocate memory resources, e.g., including VRAM, HBM, and CXL memory, based on the operational needs of the language models or neural networks. Some memory banks may be used temporarily or permanently as vector databases for retrieval-augmented generation (RAG) or storing embeddings. The memory banks may be used a carousel fashion to load data from a secondary data source, allowing layers to be loaded just in time to the memory bank. Embodiments herein may particularly be suited for high-performance computing (HPC) environments requiring efficient and reliable memory management, specifically tailored for the unique demands of LLMs and SMLs with mixed experts and / or neural layers architectures. The inclusion of CXL memory significantly enhances the system's ability to scale, manage resources dynamically, and ensure optimal performance across a wide range of existing and future computing technologies.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Examples are described in more detail below with reference to the appended drawings.

[0016] FIG. 1 illustrates an example memory management system according to some embodiments.

[0017] FIG. 2 is a flow chart of an exemplary method according to some embodiments.

[0018] FIG. 3 illustrates an example scenario.

[0019] DETAILED DESCRIPTION

[0020] Examples herein may relate to handle memory for neural networks or language models such as LLMs. A particular idea relates to that a processing unit, may be able to switch, e.g., by switching means, between using different memory banks. The memory banks may have particular memory for use by the neural networks of language models. This allows the processing unit to use more memory close to the unit without having to wait to move it from storage and can simply just switch to another bank which may be configured with needed neural layers of the neural network or the language model. When memory of a bank is not used, which can be long periods of time, the memory can power down, such as to be at an idle state, but preferably in a state where the memory is still kept intact and / or can be reset to have the same neural layers in the memory such that inference can be performed quickly when these are needed. As part of a complete system, many processing units may switch between many different memory banks.

[0021] FIG. 1 illustrates a memory management system 1 according to some examples herein. Below features of the system 1 are non-limiting, can be in isolation, and may be combined in any suitable manner. The memory management system 1 , also referred to as the system 1 , is configured for use with, e.g., configured to handle, at least one of: one or more neural networks and / or language models, e.g., LLMs or SMLs. The language models may be implemented as part of one or more neural networks. The system 1 comprises a plurality of memory banks 2, one more control units 3, one or more multiplexers and / or demultiplexers 4, and one or more processing units 5. The system 1 is configured to dynamically manage power and data flow among the memory banks 2 based on operational needs of the one or more neural networks and / or the one or more language models. The system 1 comprises switching means 6 for switching between active and inactive states of the memory banks 2. Each memory bank 2 may be a respective VRAM, DRAM, HBM, and / or CXL memory bank. When memory is discussed, any one or more of the following may be referred to: VRAM, DRAM, HBM, and CXL memory. In other words, the one or more memory banks 2 may comprise or consist of any one or more of the following banks of memory: VRAM, DRAM, HBM, CXL, or any other suitable memory technology. Whenever VRAM, HBM, or CXL is / are discussed in embodiments herein, DRAM can also be used. The language models of embodiments herein may be LLM or SML. The one or more processing units 5 may comprise or consist of any one or more of: one or more CPUs, one or more GPUs, one or more NPUs, or one or more ASICs. As used herein, ASIC may refer to transformer ASIC. The one or more control units 3 may be configured to receive one or more commands for switching the memory banks 2. The one or more commands may be received from an external source through one or more communication interfaces 7 of the system 1. The commands may be transmitted either wirelessly or through a wired connection, e.g., by the interfaces 7. The system 1 may comprise at least one storage module 8. The module 8 may be integrated within the system 1. The storage module 8 may include one or more SSDs, e.g., configured for data backup purposes. The SSDs, may be permanently fixed or removable, e.g., operatively connected to each memory bank 2 or to a main control unit of the system 1 . In embodiments herein, the main control unit may be part of the one or more control units 3. The storage module 8 may be configured to automatically backup data from the memory banks 2 periodically or in response to specific triggers such as bank switching, system shutdown, or detected faults.

[0022] The system 1 may be configured to act because a neural-network workload demands it, e.g., due to token inference, expert routing, layer call-graph. The act may be to deactivate and / or to activate memory banks 2 for the processing units 5, based on the workload demands.

[0023] The storage module 8 may be configured to restore data from the solid-state drives to the memory banks 2 upon system startup, fault recovery, or in response to a user-initiated command.

[0024] The one or more control units 3 may be configured to monitor a status of each memory bank 2, and to dynamically adjust an allocation of memory resources or the memory banks 2 based on the requirements of the one or more language models and / or the one or more neural networks. For example, when an LLM needs a set of layers, e.g., of an expert group, memory of certain banks may be allocated and activated, and other memory banks may be deactivated, e.g., for a certain processing unit of the processing units 5. The (de)activation may be performed by the switching means 6, e.g., as controlled by the one or more control units 3. The one or more multiplexers and / or demultiplexers 4 may be configured to route data between the memory banks 2 and one or more processing units 5, based on instructions received from the control unit 3.

[0025] The system 1 may comprise any one or more features according to the following System Architecture: Memory Banks, e.g., of the one or more memory banks 2: The system 1 may be configured to divide memory into one or a plurality of banks, e.g., with sizes equivalent to one or more expert models, neural layers, or complete language models LLMs or SMLs. Each expert, group of experts, neural layer, or complete language model may have a designated memory bank. These banks may be implemented using any one or more of VRAM, HBM, or CXL memory modules, typically either of them, depending on the specific performance requirements and hardware configuration. CXL memory allows for a shared and scalable memory pool accessible by multiple processing units, enhancing the system's flexibility and efficiency. Control Unit, e.g., of the one or more control units 3: may be configured to manage allocation of memory, e.g., VRAM, HBM, and CXL memory, to selected experts, neural layers, or complete language models. The control unit may dynamically adjust memory allocation based on current computational needs, leveraging the high bandwidth and low latency of CXL memory to optimize performance across various processing units, e.g., the one or more processing units 5. The control unit may be configurable through on-board settings and / or via remote commands transmitted through wired or wireless communication interfaces, e.g., the one or more interfaces 7. Multiplexer / Demultiplexer, e.g., of the one or more multiplexers / demultiplexers 4: may be configured to route data in the system 1 , e.g., between the memory banks 2, and the processing units 5, such as based on instructions from the control unit 3. The inclusion of CXL memory may allow for more efficient data routing, reducing bottlenecks and improving data transfer rates across the system 1. Processing Units, e.g., of the one or more processing units 6: may be configured to perform computations required by the one or more language models, e.g., LLMs. The processing units may access memory such as a CXL memory pool in addition to VRAM and / or HBM, e.g., of the memory banks 2. The processing units may, e.g., by the memory banks, be allowed to efficiently share memory resources and process larger datasets or models concurrently. Data Backup: Integrated SSDs, e.g., by the storage module 8, may ensure real-time data backup for each memory bank. The SSDs may be either permanently fixed or removable and are connected to the main control unit for periodic or trigger-based backups. CXL memory may be backed up or restored using these SSDs, ensuring data integrity across all memory types. Switching Mechanism, e.g., the switching means 6: The source of switching e.g., the switching means 6, may be an integrated chip on-board of the system 1 , e.g., with wired or wireless connection. The switcher, e.g., the switching means 6 or a controller thereof, may be any suitable entity such as: software, a Field Programmable Gate Array (FPGA), or an integrated circuit (IC). The switching mechanism may be configured to, e.g., by the control unit 3, to manage allocation and deallocation of memory such as of the memory banks 2, in particular CXL memory, by dynamically adjusting a memory configuration of the system 1 which may optimize performance and resource utilization of the system 1. Loading and Updating from Storage Media, e.g., the storage module 8: Memory banks 2, e.g., including those utilizing CXL memory, ma be loaded or updated from storage media such as the storage module 8, with a layer or a complete language model, e.g., LLM or SML, as needed to adapt to changing computational requirements, for example when certain memory of certain neural layers, i.e., “experts”, are needed. A high bandwidth, e.g., as in CXL, may ensure that such loading process is fast and efficient, minimizing downtime and improving system responsiveness. Temporary Storage on PCIe and CXL, e.g., of the memory banks 2: LLM models, layers, and memory may be stored temporarily on storage media connected to a PCIe slot or CXL memory of the system 1 may provide fast transfer between the memory banks 2 and the storage media. CXL memory may provide an additional layer of flexibility, allowing data to be stored and retrieved more efficiently as needed. Vector Databases for RAG or Embeddings: Some memory banks of the memory banks 2, e.g., including those implemented with CXL memory, may be used temporarily or permanently as vector databases for retrieval-augmented generation (RAG) or storing embeddings. The integration of CXL memory may allow these databases to be more accessible to multiple processing units, improving the speed and efficiency of RAG processes. Carousel Loading: Memory banks 2 e.g.,, including CXL memory banks, may be configured to use a carousel fashion to load data from a secondary data source, allowing layers to be loaded just in time to the respective memory bank. CXL's low latency may ensure that data is available when needed, reducing the risk of processing delays. Carousel loading as used herein may mean that some banks are loaded in parallel and / or when not being actively used and / or loading according to a certain order, such as in a cyclic order.

[0026] Advanced Features and Optimizations of the system 1 may comprise any one or more of: Granular Memory Allocation: The system 1 may be configured to implement fine-grained memory allocation strategies that go beyond allocating memory for whole layers or models, e.g., of the neural networks or language models. With CXL memory, the system 1 mat be configured to allocate memory resources more precisely, allowing for efficient use of memory across multiple processing units. Memory Compression: To maximize memory efficiency, the system employs memory compression techniques. Inactive models and / or layers may, e.g., by the system 1 or a unit thereof, to be compressed to reduce their memory footprint. When needed, these compressed components may be decompressed on-the-fly, balancing storage efficiency with computational readiness. Asynchronous Data Transfer: The system 1 may be configured to utilize asynchronous data transfer mechanisms, e.g., including those enabled by CXL memory, such as to overlap data loading with computation. This reduces idle times and ensures just-in-time data availability, improving overall system efficiency. Non-Volatile Memory (NVM) such as NVMe over Fabrics (NVMe-oF): The system may b configured to use NVM or NVMe-oF, e.g., to enhance data transfer speeds. This may allow SSDs, e.g., of the module 8, to connect directly to the memory banks 2, e.g., via high-speed network interfaces and / or by bypassing a processing unit 5 such as a CPU, thereby resulting in faster data transfer rates. Predictive Loading: The system 1 may be configure to incorporate machine learning (ML) models to predict which models or layers will be needed next based on current usage patterns. This predictive capability may be enhanced by CXL memory, e.g., allowing the system 1 to preload necessary data into memory banks more efficiently, reducing latency and improving responsiveness. Adaptive Caching: The system 1 may be configured to employ an adaptive caching strategy e.g., which may prioritize some memory banks, e.g., including CXL memory banks, for frequently accessed models or layers. This reduces latency for high-usage scenarios by keeping commonly used components readily available in memory. In other words, memory of some memory banks 2 may be prioritized to be caches if associated with a common used model or neural layer of a language model such as an LLM. Redundant Array of Independent Disks (RAID) Configuration for SSDs: Integrated SSDs of the system 1 may be configured in RAID setups. This configuration provides data redundancy and ensures data integrity in case of individual SSD failures, enhancing the overall reliability of the system 1. Robust Checkpointing: The system 1 may be configured to implement a robust checkpointing mechanism. The mechanism may periodically save the state of active models and layers, e.g., of the neural network or language models across the memory, e.g., of the memory banks 2. In case of system failure, this allows for quick resumption from the last checkpoint, minimizing data loss and downtime. Distributed Processing: The system 1 may be arranged to work across a cluster of GPUs, NPUs, or transformer ASICs, allowing for distributed loading and processing of models. In other words, the one or more processing units 5 may be a distributed arrangement of any one or more of: one or more GPUs, one or more NPUs, and one or more ASICs. This scalable approach enables the system 1 to handle larger models and datasets by leveraging the combined resources of multiple processing units. Networked Memory Banks: Building on the distributed processing capability, the system 1 may be configured to implement a networked approach to the memory banks 2, e.g., including those using CXL memory. Memory such as VRAM, HBM, and CXL memory from multiple processing units 5 may be arranged to be pooled and managed collectively, optimizing resource usage across the entire cluster. NVM Integration: The system 1 may be configured to integrate any suitable NVM technologies such as advanced NVM technologies, e.g., in conjunction with CXL memory. This may provide a persistent memory layer that offers higher speeds than traditional SSDs and reduces latency for memory access, bridging the gap between high-speed VRAM / HBM and slower storage. Hardware-Level Control: An input / output (I / O) pin on respective processing units 5, e.g., GPU, CPU, NPU, or transformer ASIC, may be configured as an output to control the switching of memory banks 2 e.g., including CXL memory banks. The respective I / O pin may be part of the switching means 6. The respective I / O pin may be configurable and / or controllable by the one or more control units 3. The I / O pins may allow for direct hardware-level control of memory allocation and deallocation, providing faster and more efficient memory management. Direct Memory Access (DMA): Data transfer between storage media, e.g., the module 8, and memory banks 2, e.g.,, including CXL memory banks, may be facilitated by DMA. In other words, the system may be configured to have DMA with the memory banks 2. This enables efficient, CPU- independent data loading and unloading operations, further improving performance. Unified Memory Architecture: The system 1 may be configured to dynamically switch between one or multiple memory banks 2 shared among processing units 5, e.g., CPU, GPU, NPU, and transformer ASIC, by being configured to use a unified memory architecture. In these embodiments, CXL memory may be crucial by providing a flexible, scalable memory resource that can be shared across different processing units 5. This may enable any one or more of: Seamless data sharing between different processing units 5 without explicit data transfers. Dynamic allocation and reallocation of memory resources based on the computational needs of each processing unit 5. Efficient utilization of available memory across heterogeneous computing elements. Reduced latency in data access and processing by minimizing data movement between separate memory pools. Improved performance in mixed workloads that require collaboration between CPU, GPU, NPU, and transformer ASIC. Flexible scaling of memory resources to accommodate varying demands of different Al and machine learning tasks. Transformer ASIC Integration: The system 1 may be configured to seamlessly integrate with transformer ASICs, providing optimized memory management for transformerspecific operations such as self-attention mechanisms and feed-forward networks. CXL memory may enhance this integration by offering additional memory resources that can be dynamically allocated to transformer-specific tasks, improving overall system performance. Transformer-Specific Memory Allocation: The system 1 may be configured to implement specialized allocation strategies for transformer architectures, e.g., efficiently managing memory for attention matrices, positional encodings, and intermediate activations. CXL memory may allow for more granular and flexible memory allocation, e.g., ensuring that transformer ASICs can operate at peak efficiency by minimizing memory access latencies and maximizing throughput. Multi-Head Attention Optimization: For transformer ASICs, e.g., of the processing units 5, the system 1 may be configured to provide optimized memory allocation and access patterns for multi-head attention mechanisms. CXL memory may enable parallel processing of attention heads, significantly.

[0027] Embodiments herein may relate to the system 1 and method for efficiently managing multiple banks 2, e.g., of VRAM, HBM, and CXL memory in one or more computing devices, particularly optimized for LLMs and SMLs with mixed experts and / or neural layers configurations, e.g., including specialized transformer ASICs. The one or more computing devices may be devices which is arranged with, and / or is utilizing the system 1. The system 1 may be comprise in the one or more computing devices of embodiments herein. The system 1 may be configured to dynamically allocate VRAM, HBM, and CXL memory to active expert models, neural layers, or entire language models, e.g., LLMs or SMLs, while disconnecting unused memory banks 2. These embodiments may minimize power consumption, and enhancing computational efficiency. The memory banks 2 may be loaded or updated from storage media, e.g., the module 8, with a layer or a complete language model as needed. LLM or SML models, layers, and memory may be stored temporarily on storage media, e.g., the module 8, connected to a PCIe slot or CXL memory for faster transfer between the memory banks and the storage media. The system 1 of embodiments herein may support both single token and multiple token processing and may be arranged with memory management leveraging CXL's high bandwidth and low latency to optimize resource utilization and enable concurrent processing across multiple processing units. Integrated solid-state drives (SSDs) e.g., of the module 8, may provide real-time data backup, ensuring data integrity and quick recovery. The system 1 may supports both local and remote control mechanisms, making it adaptable to a wide range of existing and future computing technologies. Embodiments herein may be for HPC environments requiring efficient and reliable memory management, e.g., specifically tailored for the unique demands of LLMs and SMLs with mixed experts and / or neural layers architectures, and applicable to systems with one or more CPUs, GPUs, NPUs, transformer ASICs, and CXL-enabled memory configurations. Embodiments herein may relate to memory management systems within computing devices, particularly to methods and systems for managing multiple banks of VRAM, HBM, and CXL memory, optimized for use with one or more central processing units CPUs, GPUs, NPUs, and transformer ASICs. For example, the banks managed may be the banks 2 of the system 1. Embodiments herein may include applications involving language models, e.g., LLMs and SMLs, featuring mixed experts and / or neural layers configurations, where efficient VRAM, HBM, and CXL memory management may be critical for performance optimization. Embodiments herein may also address dynamic allocation of memory resources across technologies mentioned herein, e.g., at least one of VRAM, HBM, and CXL, to maximize computational efficiency, scalability, and responsiveness in high-performance computing environments.

[0028] The system 1 may be configured to set different banks 2 in different sleep modes, e.g., when deactivated by the switching means 6 and / or by the one or more control units 3. Some sleep modes are discussed below. The sleep modes may be preconfigured and / or dynamically adaptable. Memory Integrity During Sleep Mode Memory management in different sleep modes may be crucial for maintaining data integrity, particularly in systems utilizing DRAM-based memory like VRAM. Light Sleep Mode: In light sleep mode, the memory bank 2 may remain intact by continuing refresh cycles at a reduced rate. The clock signal of the bank 2 may be gated or slowed, but refresh operations are maintained to ensure data retention. This allows the system 1 to save power while preserving the data stored in memory. Deep Sleep Mode (Self-Refresh Mode): In deep sleep modes, the memory of a bank 2 may enter a self-refresh mode where the memory itself manages the refresh cycles independently of a respective memory controller. This mode significantly reduces power consumption while ensuring that the data remains intact. The respective memory controller may be configured to reduce its activity, relying on the memory's internal circuits to maintain data integrity. Static random-access memory (SRAM) and Non-Volatile Memory Considerations: For SRAM, light sleep modes may reduce power but may maintain data integrity as long as the cells are still powered. In deep sleep modes, if power is cut off, data in SRAM will be lost. Non-volatile memory types e.g., Flash, may retain data without power, so sleep modes do not affect data integrity. Considerations for VRAM: Light Sleep Mode: In VRAM, a light sleep mode may maintain data integrity, with a memory controller of a bank 2 ensuring that necessary refresh cycles continue at a reduced rate, preserving the data. Deep Sleep Mode: In deeper sleep modes, VRAM may rely on self-refresh to maintain data. As long as the self-refresh mechanism is active, the memory contents remain intact. If the VRAM is completely powered down (which is rare in typical sleep modes), the data would be lost.

[0029] An example operation of the system 1 may comprise: When a decoder / router in a language model selects an expert, group of experts, neural layer, or complete language model, the corresponding memory bank(s) are enabled by allocating VRAM / HBM and disconnecting unused memory banks for token processing. Memory banks 2 may be loaded or updated from storage media, e.g., the module 8, with a layer or a complete language model (LLM or SML) as needed. LLM models, layers, and memory may be stored temporarily on storage media connected to the PCIe slot for faster transfer between the memory banks and the storage media, e.g., the module 8. The control unit 3 may be configured to monitor status of each VRAM / HBM bank 2 and may be configured to dynamically adjust the allocation of memory resources based on computational requirements. The system 1 may support both single token and multiple token processing, enabling flexible and efficient handling of language models. Some memory banks 2 may be used as vector databases for RAG or storing embeddings, either temporarily or permanently. Memory banks 2 may use a carousel fashion to load data from a secondary data source, allowing layers to be loaded just in time to the memory bank.

[0030] Advanced Features and Optimizations examples. Below follows further examples to above. Some examples may be more concrete example scenarios as discussed above and / or some examples may extend to the examples above. Granular Memory Allocation: The system 1 may allocate memory for sub-components of layers, such as individual attention heads or portions of feed-forward networks. This granular approach allows for more efficient use of VRAM and HBM, especially when only parts of a model are actively used. Memory Compression: Inactive models and layers may be compressed to reduce their memory footprint. The system 1 may balance storage efficiency with computational readiness. Asynchronous Data Transfer: The system 1 may be configured to to overlap data loading with computation, reducing idle times and ensures just-in-time data availability. NVM: To enhance data transfer speeds, the system 1 may implement NVM / NVMe-oF. This allows SSDs to connect directly to the VRAM and HBM banks via high-speed network interfaces, bypassing the CPU and resulting in faster data transfer rates. Predictive Loading: The system 1 may incorporate ML models to predict which models or layers will be needed next based on current usage patterns. This predictive capability allows the system to proactively preload necessary data into VRAM or HBM banks, reducing latency and improving responsiveness. Adaptive Caching: An adaptive caching strategy may be implemented to prioritize memory banks 2 for frequently accessed models or layers. This reduces latency for high-usage scenarios by keeping commonly used components readily available in VRAM or HBM. RAID Configuration for SSDs: The integrated SSDs may be configured in RAID (Redundant Array of Independent Disks) setups. This configuration provides data redundancy and ensures data integrity in case of individual SSD failures, enhancing the overall reliability of the system. Robust Checkpointing: The system 1 may implement a robust checkpointing mechanism that periodically saves the state of active models and layers. In case of system failure, this allows for quick resumption from the last checkpoint, minimizing data loss and downtime. Distributed Processing: The system 1 may be designed to work across a cluster of GPUs, NPUs, or transformer ASICs, allowing for distributed loading and processing of models. This scalable approach enables the system to handle larger models and datasets by leveraging the combined resources of multiple processing units. Networked Memory Banks: Building on the distributed processing capability, the system 1 may implement a networked approach to memory banks. VRAM and HBM from multiple processing units can be pooled and managed collectively, optimizing resource usage across the entire cluster. NVM Integration: The system 1 may integrate advanced non-volatile memory technologies. This provides a persistent memory layer that offers higher speeds than traditional SSDs and reduces latency for memory access, bridging the gap between high-speed VRAM / HBM and slower storage. Hardware-Level Control: An I / O pin on the GPU, CPU, NPU, or transformer ASIC may be configured as an output to control the switching of memory banks 2. This allows for direct hardware-level control of memory allocation and deallocation, providing faster and more efficient memory management. Direct Memory Access: Data transfer between storage media, e.g., the module 8, and memory banks 2, may be facilitated by Direct Memory Access (DMA). This enables efficient, CPU-independent data loading and unloading operations, further optimizing the system's performance. Unified Memory Architecture: The system 1 is configured to dynamically switch between one or multiple memory banks shared among CPU, GPU, NPU, and transformer ASIC using a unified memory architecture. This may enable: a)Seamless data sharing between different processing units without explicit data transfers, b) Dynamic allocation and reallocation of memory resources based on the computational needs of each processing unit, c) Efficient utilization of available memory across heterogeneous computing elements, d) Reduced latency in data access and processing by minimizing data movement between separate memory pools, e) Improved performance in mixed workloads that require collaboration between CPU, GPU, NPU, and transformer ASIC, f) Flexible scaling of memory resources to accommodate varying demands of different Al and machine learning tasks. Transformer ASIC Integration: The system 1 may be designed to seamlessly integrate with transformer ASICs, providing optimized memory management for transformer-specific operations such as self-attention mechanisms and feed-forward networks. This integration allows for efficient processing of transformer-based models, leveraging the specialized hardware capabilities of transformer ASICs. Transformer-Specific Memory Allocation: The memory management system 1 may implement specialized allocation strategies for transformer architectures, efficiently managing memory for attention matrices, positional encodings, and intermediate activations. This optimization ensures that transformer ASICs can operate at peak efficiency by minimizing memory access latencies and maximizing throughput. Multi-Head Attention Optimization: For transformer ASICs, the system 1 may provide optimized memory allocation and access patterns for multi-head attention mechanisms. This allows for parallel processing of attention heads, significantly improving the performance of transformer-based models. HBM-Specific Optimizations: The system 1 may implements specialized management techniques for HBM, taking advantage of its high bandwidth and 3D-stacked architecture. This includes optimized data placement strategies to maximize bandwidth utilization and minimize power consumption in HBM-equipped devices. VRAM-HBM Hybrid Management: When the system 1 is equipped with both VRAM and HBM, the memory management system may dynamically allocates workloads between these memory types based on bandwidth requirements, access patterns, and power efficiency considerations. This hybrid approach allows for optimal utilization of both memory technologies.

[0031] Embodiments herein may relate to any suitable implementations of VRAM or HBM management system, e.g., of the system 1. Specific models e.g., LLMs, may be arranged for the system 1. Architecture and methods for the system 1 may be designed to be flexible and may be adapted to work with a wide range of language models (LLMs).

[0032] FIG. 2 illustrates an example method for managing multiple memory banks 2 in one or more computing devices for use with one or more language models and / or one or more neural networks. Any one or more of the following actions may be part of the method, in any suitable order. Dashed boxes may illustrate optional actions. The method may be performed by the system 1 and / or by any suitable hardware or software unit, such as a processor or IC, of a computing device. Action 201. The method comprises dividing memory into the one or more memory banks 2. Each bank has a size equivalent to one or more expert models, neural layers, or complete language models.

[0033] Action 202. The method comprises designating separate memory banks of the one or more memory banks 2 for each expert, group of experts, neural layer, or complete language model of the one or more language models and / or the one or more neural networks. Expert as used herein may mean one or more neural layers of a neural network associated with one or more certain features.

[0034] Action 203. The method comprises, when a decoder / router associated with a language model and / or a neural network selects an expert, group of experts, neural layer, or complete language model, enabling corresponding memory banks of the one or more memory banks 2 for token processing by allocating memory resources of the corresponding memory banks and by disconnecting unused memory banks.

[0035] Action 204. The method comprises controlling the memory allocation, e.g., of action 203, through the one or more control units 3. Controlling the memory allocation may further comprises switching the banks into active or deactivated modes e.g., by the switching means.

[0036] Controlling the memory allocation may further comprise controlling unused banks 2 to be in one or more sleep modes. Controlling the memory allocation may comprise dynamically managing power and data flow among the memory banks 2 based on operational needs of the one or more neural networks and / or the one or more language models. Controlling the memory allocation comprises controlling switching between active and inactive states of the memory banks 2, e.g., by the switching means 6. Switching between the active and inactive states may comprise receiving one or more commands for switching the memory banks 2. The one or more commands may be received from an external source through the one or more communication interfaces 7 of the memory management system 1. The commands may be transmitted either wirelessly or through a wired connection.

[0037] Inactive state may be in a state with low power and / or low refresh rate but may not lose any memory.

[0038] Action 205. The method may comprise loading or updating the one or more memory banks 2 from storage media, e.g., the module 8, with a neural layer or a complete language model.

[0039] Action 206. The method may comprise monitoring a status each of the one or more memory banks 2.

[0040] Action 207. The method may comprise dynamically adjusting the allocation of memory resources of the one or more memory banks 2 based on computational requirements of the language model or neural network. Action 208. The method may comprise routing data between the one or more memory banks 2 and one or more processing units 5 based on instructions received from the one or more control units 3. The routing may be performed using a multiplexer and / or a demultiplexer such as the multiplexer and / or a demultiplexer 4.

[0041] In any one or more actions above 201-208: the one or more memory banks 2 may be comprised in the system 1 according to any embodiment herein.

[0042] FIG. 3 illustrates an example of the memory management system 1 . In the example the one or more memory banks 2 are illustrated as a first, second, and third VRAM bank, but any number of banks may apply. The VRAM banks may be identical VRAM modules wired in parallel to a GPU data bus. In this example, the GPU is the one or more processing unit 5. In this example and in any other example, each bank 2 may expose a single activation / deactivation (ACT / DEACT) pin 301 , 302, 303 or other ACT / DEACT means, illustrated by a small triangle. The ACT / DEACT means / pin may be configured to select and handle power to the memory banks 2. No data rerouting may be required. The Control Unit 3 drives the ACT / DEACT pins through the switching means 6, which may turn a bank 2, fully on or off, or alternatively into an idle mode. The idle mode may consume a limited amount of power but may save a state of the memory bank such that the memory is not lost. The GPU may be arranged to perceive the banks 2 on the same address lines such that only the banks 2 whose ACT line is high, e.g., powered on, may respond. FIG 3 may illustrate a simple general embodiment using only VRAM, parallel wiring, with only per bank activation to enable a “split layer, power gate when idle” concept. However, as discussed above, the banks 2 may use different memory technology, the GPU may be replaced by CPUs / NPUs / ASICs, etc.

[0043] In other words, the system 1 may comprises a power-gate or chip-select line, e.g., the ACT / DEACT 301 , 302, 303, for each memory bank 2, e.g., VRAM banks, that places the bank in an inactive state when de-asserted. The power-gate or chip-select line may be part of the switching means 6. De-assert as used herein may mean that a memory bank 2 is de-selected, disconnected, or deallocated, e.g., as implied when selecting and / or allocating new memory from a different memory bank 2, e.g., due to only being able to use a certain amount of memory concurrently.

[0044] In some memory memory modules such as in Double Data Rate 5 (DDR) memory modules, e.g., as part of the banks 2, the pin responsible for managing the module’s active or sleep state is typically referred to as RESET_n (Reset, active low). This pin is used to reset the memory module, effectively placing it into a known state, which can be leveraged to manage power states, including transitioning the module into a low-power or sleep mode.

[0045] Reset_n as used herein may be an alternative and / or addition to the ACT / DEACT 301 , 302, 303 and / or the control unit(s) 3 and / or the switching means 6. Additionally or alternatively, DDR5 modules incorporate a Power Management Integrated Circuit (PMIC), e.g., which may be arranged to manage power delivery and / or may influence the module’s power states. The PMIC may comprise one or more control pins and / or one or more interfaces (such as l2C) that may allow for more granular power state management, including entering and exiting sleep modes.

[0046] While RESET_n is a standard pin for managing reset and power states, the exact implementation and usage can vary depending on the specific memory module and system design.

[0047] In other words, the system 1 may comprise suitable means for resetting and / or power gating the one or more memory banks 2, e.g., when the one or more neural networks and / or the one or more language models such as an LLM is not utilizing one or more certain memory banks. Instead, during processing, one or more other banks 2 can be switched in by activating them and / or by power gating them to high power and / or resetting the memory.

[0048] Further variations and examples. Below we list a number of non-limiting examples which may be combined in any suitable manner with any of the above embodiments, e.g., as more concrete embodiments. In below examples, any entity may refer to the corresponding entity of the system 1 as describe above. For example, memory may refer to the memory banks 2. Any processing unit, e.g., CPU / GPU / ASIC / NPU may refer to the processing units 5. Any multiplexers / demultiplexers may refer to the multiplexers / demultiplexers 4, etc. The system 1 may be any suitable circuitry.

[0049] The system 1 may be configured for use with Large Language Models (LLMs) and Small Language Models (SMLs) and neural networks, comprising: one or a plurality of VRAM, HBM, or CXL banks, e.g., banks 2; one or a plurality of control units, e.g., control unit 3; one or a plurality of multiplexers / demultiplexers, e.g., multiplexers / demultiplexers 4; and one or more CPUs, GPUs, NPUs, or transformer ASICs, e.g., processing units 5. The system 1 may be configured to manage power and data flow among the VRAM, HBM, or CXL banks based on the operational needs of LLMs, SMLs, and neural networks.

[0050] The system 1 may comprise means for switching, e.g., switching means 6, between active and inactive states of VRAM, HBM, or CXL banks to optimize memory allocation for models with varying computational demands, such as those encountered in mixed experts and / or neural layers configurations, or entire language models (LLMs or SMLs), applicable to one or more CPU, GPU, NPU, and transformer ASIC architectures. The source of switching may be an integrated chip on the board itself with wired or wireless connection, and the switcher can be software, an FPGA, or an integrated IC. In other words, the means for switching, e.g., switching means 6, may be any suitable circuitry, e.g., part of a system 1 circuitry. The memory banks, including CXL memory banks, can be loaded or updated from storage media with a layer or a complete language model (LLM or SML) as needed.

[0051] The system 1 may support both single token and multiple token processing.

[0052] The control unit 3 may be configured to receive commands for switching VRAM, HBM, or CXL banks from an external source through communication interfaces, where the commands can be transmitted either wirelessly or through a wired connection, specifically for managing the allocation of VRAM, HBM, or CXL to selected experts, neural layers, or complete language models (LLMs or SMLs) in a mixed experts and / or neural layers model.

[0053] The system 1 may comprise at least one storage module 8 integrated within the system. The storage module 8 may include one or more solid-state drives (SSDs) configured for data backup purposes; The SSDs may be either permanently fixed or removable, and operatively connected to each VRAM, HBM, or CXL bank or the system's main control unit. The storage module may be configured to automatically backup data from the VRAM, HBM, or CXL banks periodically or in response to specific triggers such as bank switching, system shutdown, or detected faults.

[0054] The storage module may be configured to restore data from the solid-state drives to the VRAM, HBM, or CXL banks upon system startup, fault recovery, or in response to a user- initiated command.

[0055] The control unit may be is configured to monitor the status of each VRAM, HBM, or CXL bank and dynamically adjust the allocation of memory resources based on the computational requirements of the language model or neural network.

[0056] The multiplexer / demultiplexer is configured to route data between the VRAM, HBM, or CXL banks and the processing units (CPU, GPU, NPU, or transformer ASIC) based on the instructions received from the control unit.

[0057] The system 1 may implement granular memory allocation strategies that allocate memory at a finer granularity than whole layers or models, including sub-components of layers such as attention heads or feed-forward network chunks, across VRAM, HBM, or CXL memory banks.

[0058] The system 1 may be configured to use memory compression techniques, e.g., to reduce the footprint of inactive models and layers, decompressing them only when needed across VRAM, HBM, or CXL memory banks.

[0059] The system 1 may be configured to implement asynchronous data transfer mechanisms, e.g., to overlap data loading with computation, thereby reducing idle times and ensuring just-in- time data availability across VRAM, HBM, or CXL memory banks.

[0060] The system 1 may be configured to use NVMe over Fabrics (NVMe-oF) to connect SSDs directly to the VRAM, HBM, or CXL banks via high-speed network interfaces, e.g., bypassing the CPU for faster data transfer. The system 1 may be configured to implement ML models to predict which models or layers will be needed next based on current usage patterns, preloading necessary data to VRAM, HBM, or CXL banks proactively.

[0061] The system 1 may be configured to develop adaptive caching strategies that prioritize VRAM, HBM, or CXL memory banks for frequently accessed models or layers, reducing latency for high-usage scenarios.

[0062] SSDs of the system 1 may be configured in RAID configurations, e.g., to provide redundancy and ensure data integrity in case of SSD failures.

[0063] The system 1 may be configured to implement a robust checkpointing mechanism that periodically saves the state of active models and layers across VRAM, HBM, or CXL memory banks, e.g., allowing the system to resume from the last checkpoint in case of failure. Robust as used herein may mean that the mechanism is arranged for the system being able to roll back or restart to an earlier saved state.

[0064] The system 1 may be configured to manage VRAM, HBM, or CXL memory across a cluster of GPUs, NPUs, or transformer ASICs, allowing for distributed loading and processing of models, making the system scalable to handle larger models and datasets.

[0065] The system 1 may be configured to implement a networked approach to VRAM, HBM, or CXL memory banks where memory from multiple processing units can be pooled and managed collectively, optimizing resource usage across an entire cluster.

[0066] The system 1 may be configured to integrate NVM technologies to provide a persistent memory layer that offers higher speeds than traditional SSDs and reduces latency for memory access, in conjunction with VRAM, HBM, or CXL memory.

[0067] The system 1 may comprise an I / O pin on the GPU, CPU, NPU, or transformer ASIC which may be configured as an output to control the switching of VRAM, HBM, or CXL memory banks, allowing for direct hardware-level control of memory allocation and deallocation.

[0068] Data transfer between storage media and VRAM, HBM, or CXL memory banks of the system 1 may be arranged to be facilitated by Direct Memory Access (DMA), enabling efficient, CPU-independent data loading and unloading operations.

[0069] The system 1 may be configured to dynamically switch between one or multiple VRAM, HBM, or CXL memory banks shared among CPU, GPU, NPU, and transformer ASIC using a unified memory architecture, enabling: a) Seamless data sharing between different processing units without explicit data transfers; b) Dynamic allocation and reallocation of memory resources based on the computational needs of each processing unit; c) Efficient utilization of available memory across heterogeneous computing elements; d) Reduced latency in data access and processing by minimizing data movement between separate memory pools; e) Improved performance in mixed workloads that require collaboration between CPU, GPU, NPU, and transformer ASIC; and f) Flexible scaling of memory resources to accommodate varying demands of different Al and machine learning tasks.

[0070] The system 1 may be configured to such that the VRAM, HBM, or CXL memory banks are optimized for specific memory access patterns of transformer architectures, including efficient storage and retrieval of attention matrices and feed-forward network states.

[0071] A control unit of the system 1 , e.g., the one or more control units 3, may be configured to dynamically allocate VRAM, HBM, or CXL memory to different components of the transformer architecture, including self-attention mechanisms, feed-forward networks, and layer normalization, based on the current processing requirements.

[0072] The system 1 may be configured to optimize VRAM, HBM, or CXL memory allocation for parallel processing of multiple attention heads in transformer ASICs, enabling efficient multihead attention computations.

[0073] The system 1 may be configured to implement specialized caching strategies for transformer operations, prioritizing frequently accessed elements such as positional encodings and learned embeddings in VRAM, HBM, or CXL memory banks.

[0074] The system 1 may be configured to efficiently share memory and computational tasks with CPUs, GPUs, and NPUs through the unified memory architecture, optimizing overall system performance for transformer-based models with VRAM, HBM, or CXL memory.

[0075] The system 1 may be configured to enable scalable processing of large transformer models across multiple transformer ASICs, efficiently distributing model layers and attention mechanisms across available hardware resources, utilizing VRAM, HBM, or CXL memory banks.

[0076] The system 1 may be configured to dynamically allocate workloads between VRAM, HBM, and CXL memory based on bandwidth requirements, access patterns, and power efficiency considerations.

[0077] The system 1 may be configured to implement specialized management techniques for HBM, including optimized data placement strategies to maximize bandwidth utilization and minimize power consumption, extending these techniques to CXL memory.

[0078] A control unit of the system 1 , e.g., the one or more control units 3, may be configured to manage the allocation and deallocation of both VRAM, HBM, and CXL memory banks, optimizing the use of each memory type based on the specific requirements of different parts of the language model or neural network.

[0079] The multiplexer / demultiplexer of the system 1 , e.g., the multiplexers / demultiplexers 4 may be configured to efficiently route data between VRAM banks, HBM banks, CXL memory banks, and the processing units, minimizing data transfer latencies and maximizing overall system throughput. The system 1 may be configured to implement different caching and prefetching strategies for VRAM, HBM, and CXL memory, optimized for their respective architectures and access characteristics.

[0080] The system 1 may be configured to set different memory banks in different sleep mode, e.g., when not utilized by a neural network or language model.

[0081] Wherein in light sleep mode, a memory controller of the system 1 may be configured to continue to refresh cycles at a reduced rate to maintain data integrity across VRAM, HBM, and CXL memory banks. In a deep sleep mode, the VRAM, HBM, or CXL memory may enter a selfrefresh state, maintaining data integrity independently of the memory controller, with the capability of losing data only if the memory is completely powered down.

[0082] Wherein in deep sleep mode, the VRAM, HBM, or CXL memory may enter a self-refresh state, maintaining data integrity independently of the memory controller, with the capability of losing data only if the VRAM, HBM, or CXL memory is completely powered down.

[0083] CXL memory of the system 1 may be managed using a CXL protocol, e.g., allowing for fine-grained control over memory allocation, access, and deallocation. The CXL protocol may enable communication between the processing units (CPU, GPU, NPU, or transformer ASIC) such as the one or more processing units 5, and the CXL memory banks such as part of the memory banks 2, optimizing data transfer, memory sharing, and resource utilization in real-time based on computational demands.

[0084] The CXL protocol may allow for dynamic memory management across multiple processing units, enabling features such as hot-plugging of memory resources, memory pooling, and memory disaggregation, which enhance system scalability and flexibility.

[0085] The CXL protocol may support advanced memory features such as memory encryption, error correction, and security isolation, ensuring data integrity and security when accessing and sharing CXL memory across different processing units such as the one or more processing units 5.

[0086] CXL memory modules, e.g., of the memory banks 2, may be utilized to pool memory resources across multiple processing units such as the one or more processing untis 5, e.g., allowing for the concurrent processing of different layers, experts, or complete language models, thereby enhancing system throughput and scalability.

[0087] The CXL memory modules may support just-in-time loading of layers, experts, or complete language models from secondary storage, reducing idle times and improving system responsiveness during token processing.

[0088] The switching mechanism, e.g., the switching means 6 of the system 1 , may be configured to switch between two or more central processing units (CPUs), each having a built- in graphics processing unit (GPU) and / or neural processing unit (NPU). The system 1 may be configured to dynamically uploads different layers, models, or sub-models to the respective CPUs, allocating memory and computational tasks based on current workload requirements. The switching mechanism may ensure efficient distribution of models or layers across the CPUs, e.g., optimizing memory utilization and processing power, e.g., as in any example described above, while managing the allocation and deallocation of memory resources in line with the control mechanisms as any example described above.

[0089] The switching mechanism, e.g., the switching means 6 of the system 1 , may be configured to switch between two or more central processing units (CPUs), each having a built- in graphics processing unit (GPU) and / or neural processing unit (NPU). One of the CPUs may be configured to handle the uploading and distribution of different layers, models, or submodels to the other CPUs, dynamically allocating memory and computational tasks across the system based on workload requirements. The switching mechanism may ensure efficient distribution and synchronization of models or layers across the CPUs, e.g., optimizing memory utilization and processing power as described in any example above, while managing the allocation and deallocation of memory resources in line with the control mechanisms of any example described above.

[0090] The switching mechanism may be integrated within a memory chip of the system 1. The switching mechanism may comprise two or more memory dies inside a single package. The system 1 may be configured to enable switching between the memory dies through communication with the memory chip, wherein the intelligence required for the switching mechanism can reside either inside a memory package of the system 1 or externally, outside the memory package. This switching allows for efficient allocation of memory resources and improves performance in applications such as Artificial Intelligence (Al)inference and large language model (LLM) serving, as described in any examples above.

[0091] Additionally or alternatively, the system 1 may comprise a plurality of memory banks, e.g., banks 2, each implemented in video random-access memory (VRAM), high-bandwidth memory (HBM), or Compute Express Link (CXL)-attached memory; (b) a switch matrix, e.g., the switching means 6, configured to connect a selected memory bank to a processing unit, e.g., of the processing units 5, and to electrically isolate any non-selected memory bank; (c) a controller programmed to (i) monitor workload characteristics of a language-model inference engine, and (ii) activate only those memory banks storing parameters required for an upcoming inference step, e.g., in an LLM of the language model, while deactivating the remainder, whereby power consumption is reduced without stalling token processing.

[0092] Additionally or alternatively, the system 1 may comprise a controller configured to: (i) store, in each memory bank, e.g., of banks 2, , parameter data for a respective neural-network sub-model or layer; (ii) receive, from an inference engine, e.g., of the system 1 , an indication of which sub-model or layer is required for processing a current token or batch; and (iii) in response, couple the corresponding memory bank to the processing unit, e.g., of the units 5, and electrically isolate any non-corresponding memory bank. At least one processing unit, e.g., of the unis 5, may then execute the neural-network inference.

[0093] Additionally or alternatively, the system 1 may comprise a plurality of memory banks, e.g., the banks 2, implemented in video random-access memory (VRAM), high-bandwidth memory (HBM), or Compute-Express-Link-attached memory (CXL); a switching element, e.g., the switching means 6, coupled to the plurality of memory banks; at least one processing unit, e.g., of the units 5, that executes neural-network inference; and, a controller configured to: (i) maintain, in each memory bank, parameter data for a respective neural-network layer, expert, or sub-model; (ii) receive, from an inference engine, an indication identifying a layer, expert, or sub-model required to process a current token or batch; and (iii) in response, couple the identified memory bank to the processing unit and electrically isolate any non-identified memory bank, thereby reducing power consumption while sustaining inference throughput. Each memory bank may comprise one of: video random-access memory (VRAM), high-bandwidth memory (HBM), or Compute-Express-Link-attached (CXL) memory.

[0094] Additionally or alternatively, the memory banks 2 may be constrained by a weight matrix or attention-head chunk. Additionally or alternatively, the system may be configured to perform predictive pre-loading, e.g., of the banks 2, such as where an ML predictor as described above is used to preload the memory banks, e.g., prefetch. Additionally or alternatively, the system may be configured to perform SSD checkpointing I RAID. The system may be configured to capture persistent-store aspects as discussed above.

[0095] The method as described above may additionally or alternatively relate to any one or more of the following examples such as a method for efficiently managing multiple banks of VRAM, HBM, or CXL in computing devices for use with language models and neural networks, comprising: dividing memory into one or a plurality of VRAM, HBM, or CXL banks, each bank having a size equivalent to one or more expert models, neural layers, or complete language models (LLMs or SMLs); designating separate memory banks for each expert, group of experts, neural layer, or complete language model; enabling corresponding memory bank(s) by allocating VRAM, HBM, or CXL and disconnecting unused memory banks for token processing when a decoder / router in the language model selects an expert, group of experts, neural layer, or complete language model; and controlling the allocation of VRAM, HBM, or CXL through one or a plurality of control units, which can be configured through on-board settings or via remote commands transmitted through wired or wireless communication interfaces. The source of switching may be an integrated chip on the board itself with wired or wireless connection, and the switcher can be software, an FPGA, or an integrated IC. Memory banks e.g., banks 2, including CXL memory banks, can be loaded or updated from storage media with a layer or a complete language model (LLM or SML) as needed. The method may comprise integrating at least one storage module within the system 1 , wherein: the storage module includes one or more SSDs configured for data backup purposes; the SSDs are either permanently fixed or removable, and operatively connected to each VRAM, HBM, or CXL bank or the system's main control unit; and the storage module is configured to automatically backup data from the VRAM, HBM, or CXL banks periodically or in response to specific triggers such as bank switching, system shutdown, or detected faults. The method may comprise monitoring the status of each VRAM, HBM, or CXL bank and dynamically adjusting the allocation of memory resources based on the computational requirements of the language model or neural network.

[0096] The method may comprise routing data between the VRAM, HBM, or CXL banks and the processing units (CPU, GPU, NPU, or transformer ASIC) based on the instructions received from the control unit using a multiplexer / demultiplexer.

[0097] The method may comprise storing LLM models, layers, and memory temporarily on storage media connected to the PCIe slot or CXL memory for faster transfer between the memory banks and the storage media.

[0098] The method may comprise using some memory banks, e.g., of banks 2, including CXL memory banks, temporarily or permanently as vector databases for (RAG) or storing embeddings.

[0099] The method may comprise using some memory banks, e.g., of banks 2, including CXL memory banks, in a carousel fashion to load data from a secondary data source, allowing layers to be loaded just in time to the memory bank.

[0100] Additionally or alternatively, the method of embodiments herein may be a method for efficiently managing memory in computing devices for use with language models and neural networks, comprising: Integrating CXL memory modules, e.g., of the banks 2, that are accessible by multiple processing units, e.g., of the processing units 5, e.g., allowing for the dynamic sharing and allocation of memory resources across CPUs, GPUs, NPUs, and transformer ASICs; Utilizing CXL memory to store and manage layers, experts, or complete language models in a centralized memory pool, thereby enabling quick access and reduced data transfer times.

[0101] The method may comprise dynamically switching between VRAM, HBM, and CXL memory modules, e.g., of the memory banks 2, for loading and processing layers, experts, or complete language models, based on the current computational requirements and system conditions.

[0102] CXL memory modules, e.g., of the memory banks 2, may be used to facilitate concurrent execution of different models or layers by multiple processing units, thereby optimizing system performance and enabling scalable processing of large models. The CXL memory modules may enable asynchronous data transfer and just-in-time loading from secondary storage, reducing latency and ensuring that memory resources, e.g., of the memory banks 2, are available when needed for computational tasks.

[0103] The method may comprise utilizing the CXL memory modules, e.g., of the memory banks 2 such as to manage memory allocation at a finer granularity than whole layers or models, including sub-components of layers, experts, or models, optimizing the use of memory resources and improving computational efficiency.

[0104] The system 1 may be configured to divide layer of LLMs on the memory banks 2.

[0105] The system 1 may be configured to and / or the method may comprise, when a last layer of a first memory bank has finished processing, processing may continue with a first layer of a second memory bank. The first memory bank may be deactivated and the second memory bank may be activated, e.g., by the switching means 6.

[0106] The system 1 may be configured to and / or the method may comprise storing layers in multiple memory banks, e.g., with co-ordinated, per-layer banking and power-gating sequence driven by the model’s own execution flow.

[0107] When a first memory bank 1 is done processing, a second memory banks 2 may already be warm, e.g., activated by the control unit 3. The switch may be a few clock cycles, far faster than streaming weights from SSD / HDD.

[0108] The system 1 may be configured to and / or the method may comprise turning supply rails or refresh off, e.g., for the bank 1 after processing, thereby the energy-saved scales with time, e.g., total-bank-time - active-time.

[0109] The system 1 may be configured to and / or the method may comprise dynamic activation of on-chip SRAM banks, e.g., bank 2, for Neural Network (NN) parameters.

[0110] The system 1 may be configured to and / or the method may comprise power-state switching of the memory banks 2. Power state switching may comprise lowering the power for inactive states and / or increasing power for active states of memory banks 2.

[0111] The language model and / or NN may in embodiments herein be portioned over the memory banks 2. Each bank may dedicated to a layer group or expert, enabling per-slice activation.

[0112] Embodiments herein may explicitly covers remote I disaggregated memory, e.g., CXL memory, and / or SSD-backed banks.

Claims

CLAIMS:1 . A memory management system (1) configured for use with at least one of: one or more neural networks and / or language models, the memory management system comprising: a plurality of memory banks (2); one more control units (3); one or more multiplexers and / or demultiplexers (4); and one or more processing units (5); wherein the system (1) is configured to dynamically manage power and data flow among the memory banks (2) based on operational needs of the one or more neural networks and / or the one or more language models, wherein the system (1) comprises switching means (6) for switching between active and inactive states of the memory banks (2).

2. The system of claim 1 , wherein the control unit (3) is configured to receive one or more commands for switching the memory banks (2), the one or more commands being received from an external source through one or more communication interfaces (7) of the system (1), where the commands are transmitted either wirelessly or through a wired connection.

3. The system of claim 1 or 2, further comprising at least one storage module (8) integrated within the system (1), wherein: the storage module (8) includes one or more solid-state drives, SSDs, configured for data backup purposes; wherein the SSDs are either permanently fixed or removable, and operatively connected to each memory bank (2) or to a main control unit of the system (1); and wherein the storage module (8) is configured to automatically backup data from the memory banks (2) periodically or in response to specific triggers such as bank switching, system shutdown, or detected faults.

4. The system (1) of claim 3, wherein the storage module (8) is configured to restore data from the solid-state drives to the memory banks (2) upon system startup, fault recovery, or in response to a user-initiated command.

5. The system (1) of any of claims 1-4, wherein the control unit (3) is configured to monitor a status of each memory bank (2) and to dynamically adjust an allocation of memory resources based on the requirements of the one or more language models and / or the one or more neural networks.

6. The system of any of claims 1-5, wherein the one or more multiplexers and / or demultiplexers (4) are configured to route data between the memory banks (2) and one or more processing units (5), based on instructions received from the control unit (3).

7. A method for managing multiple memory banks (2) in one or more computing devices for use with one or more language models and / or one or more neural networks, the method comprising: dividing (201) memory into one or more memory banks (2), each bank having a size equivalent to one or more expert models, neural layers, or complete language models; designating (202) separate memory banks of the one or more memory banks (2) for each expert, group of experts, neural layer, or complete language model of the one or more language models and / or the one or more neural networks; when a decoder / router associated with a language model and / or a neural network selects an expert, group of experts, neural layer, or complete language model, enabling (203) corresponding memory banks of the one or more memory banks (2) for token processing by allocating memory resources of the corresponding memory banks and by disconnecting unused memory banks; and controlling (204) the memory allocation through a control unit (3).

8. The method of claim 7, further comprising: (205) loading or updating the one or more memory banks (2) from storage media with a neural layer or a complete language model.

9. The method of claim 7 or 8, further comprising monitoring (206) a status each of the one or more memory banks (2) and dynamically adjusting (207) the allocation of memory resources of the one or more memory banks (2) based on computational requirements of the language model or neural network.

10. The method of any of claims 7-9, further comprising routing (208) data between the one or more memory banks (2) and one or more processing units (5) based on instructions received from a control unit (3), the routing (208) is performed using a multiplexer and / or a demultiplexer.11 . The method of any of claims 7-10, wherein the one or more memory banks (2) are comprised in a memory management system (1) according to any of claims 1-6.

12. The method of any of claims 7-11 , wherein controlling (204) the memory allocation comprises dynamically managing power and data flow among the memory banks (2) based on operational needs of the one or more neural networks and / or the one or more language models.

13. The method of any of claims 7-12, wherein controlling (204) the memory allocation comprises controlling switching between active and inactive states of the memory banks (2).

14. The method of claim 13, wherein switching between the active and inactive states comprises receiving one or more commands for switching the memory banks (2), the one or more commands being received from an external source through one or more communication interfaces (7) of a memory management system (1), where the commands are transmitted either wirelessly or through a wired connection.

Images