Storing and retrieving training data for models in a data center

Abstract

Techniques for a memory controller are described. The memory controller can receive a request to store training data. The request can include a model identifier (ID) that identifies a model associated with the training data. The memory controller can send a write request to store the training data associated with the model ID in a memory region allocated for the model ID in a pooling memory. The training data stored in the memory region in the pooling memory can be addressable based on the model ID.

Description

Background Technology

[0001] Artificial intelligence (AI) can involve discovering patterns in input data, building an AI model using the patterns discovered in the input data, and using the AI ​​model to make predictions on subsequently received data. In one example, building an AI model might involve collecting input data used to generate the AI ​​model. Input data can be received from a data provider. The input data can be used as training data to train the AI ​​model. For example, training data can be used to train an AI model to recognize patterns in the input data and make inferences about the input data.

[0002] In one example, building and training an AI model can involve processing a relatively large input dataset, which can consume a relatively large amount of computing resources. Therefore, dedicated graphics processing units (GPUs) and field-programmable gate arrays (FPGAs) in cloud environments are often used to perform AI. Attached Figure Description

[0003] The features and advantages of embodiments of the present invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate the inventive features by way of example; and, wherein:

[0004] Figure 1 The applicability of AI to various industries and use cases according to exemplary embodiments is demonstrated;

[0005] Figure 2 A pooled memory subsystem including a memory controller is illustrated according to an exemplary embodiment;

[0006] Figure 3 A system for storing training data in a memory, according to an exemplary embodiment, is shown;

[0007] Figure 4 This is a flowchart illustrating operations for performing operations at a memory controller according to an exemplary embodiment; and

[0008] Figure 5 A computing system including a data storage device is shown according to an exemplary embodiment.

[0009] Reference will now be made to the exemplary embodiments shown, and they will be described herein using specific language. However, it should be understood that this is not intended to limit the scope of the invention. Detailed Implementation

[0010] Before describing the disclosed embodiments of the invention, it should be understood that this disclosure is not limited to the specific structures, process steps, or materials disclosed herein, but extends to their equivalents as will be recognized by one of ordinary skill in the art. It should also be understood that the terminology used herein is for the purpose of describing particular examples or embodiments only and is not intended to be limiting. The same reference numerals in different figures denote the same elements. The figures provided in the flowcharts and processes are provided for the purpose of clearly illustrating the steps and operations and do not necessarily indicate a particular order or sequence.

[0011] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details (e.g., examples of layouts, distances, network examples, etc.) are provided in the following description to provide a thorough understanding of various embodiments of the invention. However, those skilled in the art will recognize that such detailed embodiments do not limit the overall inventive concept set forth herein, but are merely illustrative.

[0012] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural indicators unless the context clearly indicates otherwise. Thus, for example, a reference to “bit line” includes a plurality of such bit lines.

[0013] Throughout this specification, the reference to "example" means that a particular feature, structure, or characteristic described in connection with the example is included in at least one embodiment of the invention. Therefore, the phrases "in the example" or "embodiment" appearing in various places throughout this specification do not necessarily refer to the same embodiment.

[0014] As used herein, for convenience, multiple items, structural elements, constituent elements, and / or materials may be presented in a common list. However, these lists should be interpreted as if each member in the list were individually identified as a separate and unique member. Therefore, any individual member in such a list should not be construed as a de facto equivalent of any other member in the same list solely based on its presentation in the common group, unless otherwise indicated. Additionally, various embodiments and examples of the invention, as well as alternatives to its various components, may be referred to herein. It should be understood that such embodiments, examples, and alternatives should not be construed as de facto equivalents of each other, but should be considered as separate and autonomous representations within the scope of this disclosure.

[0015] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details (e.g., examples of layouts, distances, network examples, etc.) are provided in the following description to provide a thorough understanding of embodiments of the invention. However, those skilled in the art will recognize that the technique can be practiced without one or more of the specific details or with other methods, components, layouts, etc. In other instances, well-known structures, materials, or operations may not have been shown or described in detail to avoid obscuring aspects of this disclosure.

[0016] In this disclosure, terms such as “comprises,” “comprising,” “containing,” and “having” may have the meanings they are given under U.S. Patent Law and may mean “includes,” “including,” etc., and are generally interpreted as open-ended terms. The terms “consisting of” or “consists of” are closed-ended terms and include only the components, structures, steps, etc., specifically listed in conjunction with such terms, and those pursuant to U.S. Patent Law. “consisting essentially of” or “consists essentially of” have the meanings they are generally given under U.S. Patent Law. Specifically, such terms are generally closed-ended terms, with exceptions allowing the inclusion of additional items, materials, components, steps, or elements that do not materially affect the essential and novel characteristics or function of the item used in conjunction with them. For example, if a trace element is present under the language of “consisting essentially of,” its presence in the composition without affecting the nature or properties of the composition is permissible, even if not explicitly stated in the list of items following such a term. When open-ended terms (such as “including” or “comprise”) are used in this specification, it should be understood and should be understood that direct support is given to the language of “consistently made of” and the language of “made of”, as expressly stated, and vice versa.

[0017] The terms “first,” “second,” “third,” “fourth,” etc., as used in the specification and claims (if any) are used to distinguish between similar elements and are not necessarily used to describe a particular order or chronological sequence. It should be understood that any terms thus used are interchangeable where appropriate, so that the embodiments described herein can, for example, operate in an order different from that shown or otherwise described herein. Similarly, if a method is described herein as comprising a series of steps, the order of such steps as presented herein is not necessarily the only order in which such steps can be performed, and some said steps may be omitted and / or other steps not described herein may be added to the method.

[0018] As used herein, comparative terms such as “increased,” “decreased,” “better,” “worse,” “higher,” “lower,” and “enhanced” refer to attributes of devices, components, or activities that are measurably different from other devices, components, or activities in surrounding or adjacent areas, in a single device or among multiple similar devices, in a group or class, in multiple groups or classes, or compared to existing technology. For example, a data area with an “increased” risk of damage can refer to an area of ​​a memory device that is more likely to have write errors compared to other areas in the same memory device. Several factors can lead to such an increased risk, including location, manufacturing process, number of program pulses applied to the area, etc.

[0019] As used herein, the term "substantially" refers to the degree or level of completeness or near-completeness of an action, characteristic, attribute, state, structure, item, or result. For example, a "substantially" closed object would mean that the object is completely or almost completely closed. In some cases, the exact permissible deviation from absolute completeness can depend on the specific context. However, generally speaking, a closeness to completeness will have the same overall result as obtaining absolute and totality. When used in a negative sense, the use of "substantially" is equivalent to referring to the complete or near-complete absence of an action, characteristic, attribute, state, structure, item, or result. For example, a composition "substantially free of" particles is completely free of particles, or so close to being completely free of particles that the effect is the same as if it were completely free of particles. In other words, a composition "substantially free of" an ingredient or element may still actually contain such a substance (as long as its measurable effect is absent).

[0020] As used herein, the term “about” is used to provide flexibility toward the endpoints of a numerical range by indicating that a given value may be “slightly above” or “slightly below” the endpoints. However, it should be understood that even when the term “about” is used in conjunction with specific numerical values ​​in this specification, support for precise numerical values ​​is provided in addition to the term “about”.

[0021] Data volume and data may be expressed or presented in range format herein. It should be understood that such range format is used merely for convenience and brevity, and therefore should be flexibly interpreted to include not only the values ​​explicitly stated as range boundaries, but also all individual values ​​or subranges contained within that range, as if each value and subrange were explicitly stated. For example, the numerical range “about 1 to about 5” should be interpreted to include not only the explicitly stated values ​​of about 1 to about 5, but also the individual values ​​and subranges within the indicated range. Thus, included within this numerical range are individual values ​​(e.g., 2, 3, and 4) and subranges (e.g., 1-3, 2-4, and 3-5, etc.), and individually 1, 1.5, 2, 2.3, 3, 3.8, 4, 4.6, 5, and 5.1.

[0022] This same principle applies to ranges where only one value is recorded as the minimum or maximum. Furthermore, this interpretation should apply regardless of the width of the range or the characteristic being described.

[0023] The following provides an initial overview of technical embodiments, followed by a more detailed description of specific technical embodiments later. This initial overview is intended to help the reader understand the technology more quickly and is not intended to identify key or fundamental technical features, nor to limit the scope of the claimed subject matter. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0024] In recent years, the increased performance and functionality of hardware platforms have enabled the development of artificial intelligence (AI). Recent advances in AI are likely due to high-density computing platforms, which can be better equipped to handle increasing dataset sizes. In other words, these high-density computing platforms can achieve increased performance levels on AI workloads. For example, since training AI models (or deep learning networks) involves moving large amounts of data, current hardware platforms for AI can include high-capacity, high-speed, high-bandwidth memory technologies that can provide maximum levels of on-chip storage and increased memory access speeds. Current hardware platforms for AI can provide separate pipelines for computation and data management. Furthermore, current hardware platforms for AI can include bidirectional high-bandwidth links that enable application-specific integrated circuits (ASICs) to interconnect, allowing data to move between them. This can result in additional computing resources being allocated to tasks or model size scaling without slowing down the process.

[0025] In one example, training an AI model can involve massive amounts of data processing. Relatively large amounts of computing power and memory capacity can be used to generate AI models, such as deep neural networks. Within this large amount of data (e.g., multiple different datasets), data priorities and values ​​can vary depending on the data being used. For example, a particular dataset that is beneficial when training a first AI model may be less beneficial when training a second AI model. However, in past solutions, identifying and selecting higher-priority datasets across multiple different datasets would be difficult. As a result, all the different datasets would be used to train the AI ​​model, increasing computational and memory resources, even when some datasets have a smaller impact during the training of the AI ​​model.

[0026] Therefore, in this technique, different datasets can be associated with specific AI models (or AI model identifiers (IDs)). Datasets associated with a specific AI model can be high-priority datasets for that AI model, while datasets not associated with a specific AI model can be low-priority datasets for that AI model. These datasets can be configured on a per-dataset basis so that individual datasets can be associated with specific AI models. As a result, high-priority datasets can be used to train one or more AI models, while low-priority data can be excluded from training those models, thereby reducing computational and memory consumption.

[0027] In this technology, a memory controller can initiate read and write operations in pooled memory. The memory controller and pooled memory can be included in a pooled memory subsystem. In one example, the pooled memory subsystem can be included in a storage rack in a data center. While the following extensive discussion involves storage racks, drawers, and data centers, this is merely for convenience and should not be considered limiting. Pooled memory can provide the benefit of dynamically scaling up memory attached to a specific AI model running on a particular node, which can improve the total cost of ownership (TCO). In one example, the memory controller can receive read and write requests from requesters (e.g., processors, cores, nodes, hosts, or AI hardware platforms), and the memory controller can initiate read and write operations against the requesters. For example, the memory controller can initiate read and write operations by generating memory commands (e.g., read requests, write requests, or erase requests) sent to the media controller on a dual in-line memory module (DIMM). The processor and AI hardware platform, as well as the pooled memory, can be included in a separate drawer within the pooled memory subsystem. In one example, in response to a write request from the processor, the memory controller can write training data to the pooled memory with respect to a specific AI model identifier (ID) (or AI model). In other words, the training data to be stored in the pooled memory can be associated with a specific AI model ID. In another example, in response to a read request from the AI ​​hardware platform, the memory controller can read training data from the pooled memory. The memory controller can read the training data according to the AI ​​model ID, rather than based on a specific address in the pooled memory. Furthermore, the AI ​​model ID can be a universally unique identifier (UUID), and the AI ​​model ID can be previously agreed upon by the device and data center or owner model.

[0028] In one example, the memory in the pooled memory subsystem can be volatile memory, non-volatile memory (NVM), or a combination thereof. Volatile memory can include any type of volatile memory and is not considered to be limiting. Volatile memory is a storage medium that requires power to maintain the state of the data stored by the medium. Non-limiting examples of volatile memory can include random access memory (RAM), such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), etc., including combinations thereof. SDRAM memory can include any variant thereof, such as single data rate SDRAM (SDR DRAM), double data rate (DDR) SDRAM (including DDR, DDR2, DDR3, DDR4, DDR5, etc., collectively referred to as DDRx), and low power DDR (LPDDR) SDRAM (including LPDDR, LPDDR2, LPDDR3, LPDDR4, etc., collectively referred to as LPDDRx). In some examples, the DRAM conforms to standards issued by JEDEC, such as JESD79F for DDR SDRAM, JESD79-2F for DDR2 SDRAM, JESD79-3F for DDR3 SDRAM, JESD79-4A for DDR4 SDRAM, JESD209B for LPDDR SDRAM, JESD209-2F for LPDDR2 SDRAM, JESD209-3C for LPDDR3 SDRAM, and JESD209-4A for LPDDR4 SDRAM (these standards are available at www.jedec.org; DDR5 SDRAM is coming soon). Such standards (and similar standards) can be referred to as DDR-based or LPDDR-based standards, and the communication interfaces implementing such standards can be referred to as DDR-based or LPDDR-based interfaces. In one specific example, the system memory can be DRAM. In another specific example, the system memory can be DDRx SDRAM. In yet another specific aspect, the system memory can be LPDDRx SDRAM.

[0029] NVMs are storage media that do not require power to maintain the state of the data stored on them. While NVMs have traditionally been used for data storage or long-term persistent storage tasks, new and evolving memory technologies allow NVMs to be used for roles beyond traditional data storage. One example of such a role is using NVMs as main memory or system memory. Non-volatile system memory (NVMsys) combines the data reliability of traditional storage with ultra-low latency and high bandwidth performance to offer many advantages over traditional volatile memory, such as high density, large capacity, lower power consumption, and reduced manufacturing complexity. For example, byte-addressable in-place write NVMs (e.g., three-dimensional (3D) cross-point memory) can act as byte-addressable memory similar to dynamic random access memory (DRAM) or as block-addressable memory similar to NAND flash memory. In other words, such NVMs can act as system memory or as persistent memory (NVMstor). In some cases where NVMs are used as system memory, the stored data can be discarded or otherwise rendered unreadable when the power to the NVMsys is interrupted. NVMsys also allows for increased flexibility in data management by providing non-volatile, low-latency memory that can reside closer to the processor in the computing device. In some examples, NVMsys can reside on the DRAM bus, enabling it to provide ultra-fast DRAM-class access to data. NVMsys is also useful in computing environments that frequently access large, complex datasets, and in environments sensitive to downtime caused by power failures or system crashes.

[0030] Non-limiting examples of NVMs may include planar or three-dimensional (3D) NAND flash memory, including single-threshold or multi-threshold NAND flash memory, NOR flash memory, single-level or multi-level phase change memory (PCM) (e.g., chalcogenide glass PCM, planar or 3D PCM), cross-point array memory (including 3D cross-point memory, memory based on non-volatile dual in-line memory modules (NVDIMM), such as flash memory-based memory (NVDIMM-F), flash memory / DRAM-based memory (NVDIMM-N), persistent memory-based memory (NVDIMM-P), 3D cross-point based NVDIMM memory), and resistive RAM (ReRAM) (including metal oxide-based or oxygen vacancy-based ReRAM, such as HfO2, Hf / HfO). x Ti / HfO2, TiO x and TaO xReRAM; filament-based ReRAM, such as Ag / GeS2, ZrTe / Al2O3, and Ag-based ReRAM; programmable metallized cell (PMC) memories, such as conductive bridged RAM (CBRAM), silicon-oxide-nitride-oxide-silicon (SONOS) memories, ferroelectric RAM (FeRAM), ferroelectric transistor RAM (Fe-TRAM), antiferroelectric memories, polymer memories (e.g., ferroelectric polymer memories), magnetoresistive RAM (MRAM), write-in-place nonvolatile MRAM (NVMRAM), spin-transfer torque (STT) memories, spin-orbit torque (SOT) memories, nanowire memories, electrically erasable programmable read-only memories (EEPROM), nanotube RAM (NRAM), other memristor and thyristor-based memories, spintronic junction-based memories, magnetic tunnel junction (MTJ)-based memories, domain wall (DW)-based memories, and combinations thereof. The term "memory device" can refer to the die itself and / or the packaged memory product. NVM can be byte- or block-addressable. In some examples, NVM can conform to one or more standards issued by the Joint Electronic Equipment Committee (JEDEC), such as JESD21-C, JESD218, JESD219, JESD220-1, JESD223B, JESD223-1, or other suitable standards (the JEDEC standards described herein are available at www.jedec.org). In one specific example, NVM can be 3D crosspoint memory. In another specific example, the memory can be NAND or 3D NAND memory. In yet another specific example, the system memory can be STT memory.

[0031] Figure 1 Examples of AI applicability across various industries and use cases are illustrated. For instance, AI has broad potential applicability in a wide range of fields, such as agriculture (e.g., for personalizing crops to individual conditions), finance (e.g., for identifying fraudulent transactions or personalizing financial products), pharmaceuticals (e.g., for optimizing clinical trials), the automotive industry (e.g., for identifying and navigating roads), healthcare (e.g., for predicting personalized health outcomes or diagnosing diseases), the public / social sector, consumer goods (e.g., for optimizing sales strategies), manufacturing (e.g., for predictive maintenance), telecommunications, energy (e.g., for predictive maintenance), media (e.g., for personalized advertising or discovering new consumer trends), travel, transportation, and logistics (e.g., for optimizing pricing and scheduling in real time), and more. Furthermore, certain industries and use cases can be correlated with an increased impact score to indicate where AI applicability is more significant.

[0032] In one configuration, the memory pooling and de-aggregation architecture of this technology can be used in a data center architecture that hosts AI data for training and inference of multiple AI models being trained. The memory pooling and de-aggregation architecture allows for handling various data priorities at the per-AI-model granularity and can be optimized to handle AI applications (e.g., specifying a Service Level Agreement (SLA) or satisfying Quality of Service (QoS)). In one example, QoS may involve limiting the amount of bandwidth used to send data to a specific AI model ID, or QoS may provide additional priorities for certain types of AI models or AI model IDs. The memory pooling and de-aggregation architecture can manage and schedule pooled memory blocks or data units (e.g., memory rows or 4-kilobyte (KB) data blocks) based on the AI ​​model and priority associated with the pooled memory block or data unit, which can allow for improved data management of data center equipment for AI devices.

[0033] In current technology, memory pooling and de-aggregation architectures can implement training dataset memory pools, and these novel memory pools can store and organize training data on a model-by-model basis. In one example, the memory pool can consist of multiple de-aggregated memory devices. The training dataset memory pool can be dedicated to training AI models. As a result, the software stack can access the training data in the pooled memory on a per-AI-model basis. Furthermore, performance and reliability requirements can be specified per AI model, allowing for prioritization on a per-dataset basis.

[0034] In current technologies, memory pooling and de-aggregation architectures may include hierarchical memory controllers that can distribute training data across a number of memory drawers depending on the requirements of the AI ​​model. The hierarchical memory controller can determine which memory drawers and racks to distribute training data to meet SLAs (e.g., specific bandwidths) and can distribute training data across multiple AI data drawers (e.g., N drawers with different resilience characteristics, where N is an integer).

[0035] Memory pooling and de-aggregation architectures may include rack-pooled memory controllers. A given storage rack may include one or more rack-pooled memory controllers, depending on the performance level required by the storage rack. Rack-pooled memory controllers can define specific performance and resilience levels for particular AI models. Rack-pooled memory controllers can receive write requests (e.g., from processors, nodes, hosts, cores, etc.), and may distribute training data across different memory pools depending on application performance requirements (e.g., increased application performance requirements may mean more distribution of training data). As described in more detail below, rack-pooled memory controllers may include interfaces for registering new types of AI models. The interface can provide an AI model ID and metadata including the definition of a dataset entry (e.g., the size of the dataset entry). Rack-pooled memory controllers may include interfaces for allocating new memory to a specific AI model ID. Rack-pooled memory controllers may determine the allocation of memory in existing memory pools based on performance requirements (e.g., SLAs). Rack-pooled memory controllers may include interfaces for reading and writing training datasets for a specific AI model ID. A rack-pooled memory controller can determine the appropriate memory pool for reading and writing training datasets to optimize performance and / or meet SLAs. Additionally, the rack-pooled memory controller may include an interface for allowing other compute elements to discover the training dataset (corresponding to the AI ​​model ID) and associated metadata currently hosted in the memory pooling and de-aggregation architecture.

[0036] In current technology, AI training models running on AI hardware platforms can pull training data from pooled memory for a specific AI model ID, rather than pulse-based data. For example, a rack-pooled memory controller can use metadata defining the specific AI model's dataset structure to collect memory lines from endpoint dual in-line memory modules (DIMMs), and the controller can send data corresponding to the memory lines to the AI ​​training model. For instance, instead of reading an address X known to contain session 1, the AI ​​training model can pull a dataset for AI model session type 1, providing additional flexibility and ease of distributing and managing training data across multiple memory pools. In other words, unlike traditional memory lines, training data can be accessed according to the AI ​​dataset instance ID.

[0037] The memory pooling and de-aggregation architecture may include a drawer memory controller, which may provide an interface for writing and reading training datasets per AI model. The write interface may originate directly from the client or from the rack-pooled memory controller, depending on latency requirements. For example, a write interface from the rack-pooled memory controller may have higher latency but higher bandwidth due to load balancing. Similar to the rack-pooled memory controller, the drawer memory controller may have interfaces for performing the following operations: registering new types of AI models, allocating new memory to a specific AI model ID, reading and writing training datasets for a specific AI model ID, and discovering training datasets currently being hosted in the memory pooling and de-aggregation architecture (corresponding to the AI ​​model ID). The drawer memory controller may perform load balancing within DIMMS (or other types of memory) hosted in the same memory pool. The drawer memory controller may include logic for reading training data from the DIMM using metadata for a specific AI model ID (which may indicate how the training dataset is constructed). Additionally, after reading the training dataset, the drawer memory controller may remove the training dataset from memory based on bits in a read request specifying the removal of the training dataset from memory.

[0038] Figure 2 An example of a pooled memory subsystem 200 is shown, which includes a rack-based pooled AI model memory controller 230 (or memory controller). In one example, the pooled memory subsystem 200 may be included in a storage rack. The storage rack may be included in a data center, and the storage rack may be one of multiple storage racks in the data center. The pooled memory subsystem 200 may include multiple drawers, such as a first drawer 210 (drawer 1) and an Nth drawer 220 (drawer N). Each drawer may include a processor, an AI hardware platform, a network interface, a drawer memory controller, pooled memory, etc.

[0039] In one example, the first drawer 210 may include a first processor 214 and a second processor 216, which can be connected to a network interface 212. The first processor 214 and the second processor 216 can communicate with a rack-pooled AI model memory controller 230. Furthermore, the first processor 214 may include a first pooled memory interface 215, and the second processor 216 may include a second pooled memory interface 217. The first pooled memory interface 215 and the second pooled memory interface 217 can communicate with drawer AI model memory controllers 240 and 250 (or auxiliary memory controllers in the pooled memory subsystem), which can access pooled memory 242 and pooled memory 252, respectively.

[0040] In one example, the Nth drawer 220 may include a first AI hardware platform 224 and a second AI hardware platform 226, which can be connected to a network interface 222. The first AI hardware platform 224 and the second AI hardware platform 226 can communicate with a rack-pooled AI model memory controller 230. Furthermore, the first AI hardware platform 224 may include a third pooled memory interface 225, and the second AI hardware platform 226 may include a fourth pooled memory interface 227. The first AI hardware platform 224 and the second AI hardware platform 226 can communicate with drawer AI model memory controllers 240 and 250, which can access pooled memory 242 and pooled memory 252, respectively.

[0041] In one example, access to the pooled memory 242 and the pooled memory 252 can be achieved via network interface 212 in the first drawer 210 and network interface 222 in the Nth drawer 220, respectively.

[0042] In one configuration, rack-pooled AI model memory controller 230 may receive a request from first processor 214 (or alternatively, from a host, node, or core) to store training data in pooled memory subsystem 200. For example, the request may be to store training data in pooled memory (e.g., pooled memory 242 or pooled memory 252) within pooled memory subsystem 200. The request may include training data to be stored and an AI model ID identifying the AI ​​model associated with the training data. In other words, the training data to be stored in pooled memory subsystem 200 can be used to train a corresponding AI model (with an AI model ID) in pooled memory subsystem 200. The AI ​​model may be a training or inference AI model. In one example, upon receiving the request, rack-pooled AI model memory controller 230 may initiate a storage operation (e.g., send a write request) to store the training data associated with the model ID in pooled memory subsystem 200. More specifically, the rack-pooled AI model memory controller 230 can store training data in memory areas allocated for model IDs within pooled memories 242, 252.

[0043] In one example, the rack-pooled AI model memory controller 230 can store training data in memory areas of pooled memories 242 and 252 via drawer AI model memory controllers 240 and 250. In other words, the rack-pooled AI model memory controller 230 may not store training data directly in pooled memories 242 and 252, but rather indirectly via drawer AI model memory controllers 240 and 250. Furthermore, pooled memories 242 and 252 may expose interfaces for increasing or decreasing the amount of memory dedicated to a specific AI model ID. The ability of pooled memories 242 and 252 to dynamically and autonomously increase or decrease the amount of memory dedicated to a specific AI model ID can be an improvement over previous solutions. Otherwise, the software stack would undesirably use the interface for increasing or decreasing the amount of pooled memory.

[0044] As a non-limiting example, the first processor 214 may be a data provider that receives vehicle sensor data, for example, from an Internet of Things (IoT) device. The first processor 214 may provide the vehicle sensor data for storage in pooled memories 242, 252 within the pooled memory subsystem 200. The vehicle sensor data may be associated with a vehicle AI model (corresponding to a vehicle AI model ID) operating in the pooled memory subsystem 200. The first processor 214 may send a storage request (or push data command) including the vehicle sensor data and instructions to the vehicle AI model. In response to receiving a storage request, the rack-pooled AI model memory controller 230 may store the vehicle sensor data in a memory area allocated for the vehicle AI model ID within the pooled memories 242, 252. In other words, the memory area may have been previously allocated for the vehicle sensor data associated with the vehicle AI model ID. When the rack-pooled AI model memory controller 230 receives a request to store the vehicle sensor data, it may store the vehicle sensor data in the previously allocated memory area within the pooled memories 242, 252.

[0045] In one configuration, the rack-pooled AI model memory controller 230 can receive a request from a first AI hardware platform 224 running an AI model. The request may be to read training data from memory regions in pooled memories 242, 252 allocated for an AI model ID (corresponding to the AI ​​model). In one example, after receiving the request, the rack-pooled AI model memory controller 230 may initiate a read operation (e.g., send a read request) to read training data from memory regions in pooled memories 242, 252. Alternatively, the rack-pooled AI model memory controller 230 may read training data from memory regions based on the AI ​​model ID. The rack-pooled AI model memory controller 230 may provide the retrieved training data to the first AI hardware platform 224 running the AI ​​model, and the training data may be used to train the AI ​​model at the first AI hardware platform 224.

[0046] In one example, the rack-pooled AI model memory controller 230 can read training data from the memory regions of pooled memories 242 and 252 via drawer AI model memory controllers 240 and 250. In other words, the rack-pooled AI model memory controller 230 can read training data indirectly from pooled memories 242 and 252 via drawer AI model memory controllers 240 and 250, rather than directly from pooled memories 242 and 252.

[0047] As a non-limiting example, the first AI hardware platform 224 can be a data consumer that consumes vehicle sensor data when training a vehicle AI model running on the first AI hardware platform 224. The first AI hardware platform 224 can consume vehicle sensor data stored in the pooled memory subsystem 200. For example, the first AI hardware platform 224 can send a read request (or pull data command) including an instruction for the vehicle AI model. In response to receiving the read request, the rack-pooled AI model memory controller 230 can read the vehicle sensor data from the memory regions allocated for the vehicle AI model ID in the pooled memories 242, 252. In other words, based on the vehicle AI model ID included in the read request, the rack-pooled AI model memory controller 230 can identify the memory regions in the pooled memories 242, 252 allocated for the vehicle AI model ID, and then read the vehicle sensor data from the memory regions. The rack-pooled AI model memory controller 230 can provide retrieved vehicle sensor data to the first AI hardware platform 224, and the first AI hardware platform 224 can consume the vehicle sensor data for training a vehicle AI model running on the first AI hardware platform 224.

[0048] In one example, rack-pooled AI model memory controller 230 may receive a request from second processor 216 to read training data from memory regions allocated for (corresponding to AI model) AI model IDs in pooled memories 242, 252. In one example, after receiving the request, rack-pooled AI model memory controller 230 may initiate a read operation to read training data from the memory regions in pooled memories 242, 252. Rack-pooled AI model memory controller 230 may read training data from the memory regions based on AI model IDs. Rack-pooled AI model memory controller 230 may provide the retrieved training data to second processor 216. Second processor 216 may store the training data in local memory or a cache configured to store training data per AI model ID, and the training data may be accessible to applications running on second processor 216.

[0049] In one example, the rack-pooled AI model memory controller 230 can store and read training data in memory regions of pooled memories 242 and 252 on a per-model-ID basis. In this example, the rack-pooled AI model memory controller 230 can write and read training datasets for specific AI model IDs (corresponding to AI models). In other words, the training data written to or read from memory regions of pooled memories 242 and 252 can be addressable based on model ID.

[0050] Alternatively, the training data written to or read from memory regions in pooled memories 242, 252 may be addressable based on the AI ​​model type of the AI ​​model associated with the training data. For example, rack-pooled AI model memory controller 230 may receive a read command for reading training data associated with a vehicle AI model type (and there may be multiple AI model IDs corresponding to the vehicle AI model type). Therefore, rack-pooled AI model memory controller 230 can access the training data based on the AI ​​model type of the AI ​​model associated with the training data.

[0051] In the previous solution, memory lines were used to address memory. For example, a memory line could be 64 bytes of data. Therefore, in the previous solution, when the processor performed an operation in memory (e.g., a read or write operation), the processor knew the physical memory address of the specific data to be accessed. The processor retrieved the data from the physical memory address, and the data was placed in a processor register and made accessible to the application. In the current solution, instead of using physical memory addresses to address pooled memories 242 and 252, the AI ​​model IDs are used to address pooled memories 242 and 252.

[0052] In one configuration, the rack-pooled AI model memory controller 230 may receive a request from the first processor 214 to register an AI model ID (associated with the AI ​​model). At this time, when the request is received, there is no allocation of training data associated with the AI ​​model ID in the pooled memories 242, 252 of the pooled memory subsystem 200. The request received from the first processor 214 may include: the AI ​​model ID associated with the AI ​​model, the desired Quality of Service (QoS) for the AI ​​model as defined in a Service Level Agreement (SLA), the resilience level of the AI ​​model, and the memory capacity for storing the training data for the AI ​​model (e.g., the amount of memory to be allocated for storing the training data for the AI ​​model). In one example, the request may be a register model command that includes the AI ​​model ID, resilience level, and desired QoS. In response to receiving a request, the rack-pooled AI model memory controller 230 can register an AI model ID, which may involve allocating memory regions in pooled memories 242, 252 for storing training data associated with the AI ​​model ID, wherein the allocation of memory regions may correspond to the memory capacity included in the request received from the first processor 214.

[0053] In one example, drawers in the pooled memory subsystem 200 (e.g., first drawer 210 (drawer 1) and Nth drawer 220 (drawer N)) may include memory extensions, which may include pooled memories 242, 252. Depending on the individual requirements of the AI ​​model (in terms of memory), different portions of the pooled memories 242, 252 may be allocated for different AI models. More specifically, a portion of the memory in the pooled memories 242, 252 may be allocated for a specific AI model ID. The portion of memory may be allocated based on a training dataset of a specified size for that AI model ID. As a result, the training data associated with that AI model ID may be stored in the previously allocated portion of memory in the pooled memories 242, 252 (i.e., in the portion of memory specifically allocated for that type of training data). After allocating a portion of memory for the AI ​​model ID, the training data for the AI ​​model ID stored in the portion of memory can be accessed using the AI ​​model ID (or dataset ID), rather than the physical memory address of the memory in the pooled memories 242, 252.

[0054] As an example, the rack-pooled AI model memory controller 230 can receive a request to read training data associated with AI model ID 100 or dataset ID 15 of AI model ID 100, and in response, the rack-pooled AI model memory controller 230 can retrieve the requested training data from portions of memory allocated in pooled memories 242, 252 for AI model ID 100. The rack-pooled AI model memory controller 230 can return the training data associated with AI model ID 100 or dataset ID 15 of AI model ID 100.

[0055] In one example, the rack-pooled AI model memory controller 230 can store QoS policies 232 and resilience policies 234 for multiple AI model IDs. When a specific AI model ID is registered at the rack-pooled AI model memory controller 230, QoS policies 232 and resilience policies 234 can be defined. QoS policy 232 can indicate the desired QoS for a specific AI model ID. For example, training data for a specific AI model ID can be written to or read from pooled memories 242, 252 based on the bandwidth defined by QoS policy 232 for that specific AI model ID. In other words, a certain amount of bandwidth can be provided for the training data stream for that specific AI model ID, which conforms to QoS policy 232 for that specific AI model ID. Furthermore, QoS policy 232 can be bandwidth-based on SLAs or per-stream-based on priority. Resilience policy 234 can indicate the desired resilience level for a specific AI model ID. For example, a specific model ID can have a desired resilience level of 90%. Different pooling memories 242 and 252 can have different elasticity properties or characteristics. Therefore, as an example, the rack-pooled AI model memory controller 230 can allocate a first memory region in the pooling memory 242 and a second memory region (which can be in different drawers) in the pooling memory 252, which allows training data to be stored and retrieved according to the desired elasticity level.

[0056] In one configuration, the rack-pooled AI model memory controller 230 can send discovery information to the first processor 214 and / or the first AI hardware platform 224 in the pooled memory subsystem 200. For example, the rack-pooled AI model memory controller 230 can receive a request for discovery available data for a specific AI model ID from the first processor 214 and / or the first AI hardware platform 224, and in response, the rack-pooled AI model memory controller 230 can send discovery information for that specific AI model ID. In other words, the discovery information enables the first processor 214 and / or the first AI hardware platform 224 to discover available training data for each AI model ID in the pooled memories 242, 252 in the pooled memory subsystem 200.

[0057] As a non-limiting example, the first AI hardware platform 224 can host a fraud detection AI model. The first AI hardware platform 224 may wish to obtain fraud detection training data stored in pooled memories 242 and 252, which can be consumed by the first AI hardware platform 224 to train the fraud detection AI model. The first AI hardware platform 224 may know the fraud detection AI model ID associated with the fraud detection training data, but initially may not know whether the fraud detection training data associated with the fraud detection AI model ID is being stored in pooled memories 242 and 252. The first AI hardware platform 224 can send a request for discovery available data for the fraud detection AI model ID to the rack-pooled AI model memory controller 230. In response, the rack-pooled AI model memory controller 230 can provide discovery information indicating that the fraud detection training data corresponding to the fraud detection AI model ID is currently stored in pooled memories 242 and 252. Based on this discovery information, the first AI hardware platform 224 can send a read request for the fraud detection AI model ID to read the fraud detection training data from pooled memories 242 and 252.

[0058] In one configuration, rack-pooled AI model memory controller 230 may receive a request from first processor 214 to store training data in pooled memories 242, 252. The request may include training data to be stored and an AI model ID that identifies the AI ​​model associated with the training data. Rack-pooled AI model memory controller 230 may initiate a storage operation to store training data associated with the AI ​​model ID across multiple memory regions in pooled memories 242, 252 of pooled memory subsystem 200. Rack-pooled AI model memory controller 230 may distribute training data across multiple memory regions to meet the SLA for the AI ​​model ID. For example, distributing training data across multiple memory regions can achieve the desired storage time according to the SLA for the AI ​​model ID, and this desired storage time may not be achieved by writing training data to a single memory region in a single pooled memory. Additionally, rack-pooled AI model memory controller 230 may initiate a read operation to read training data from one or more of the multiple memory regions, and may read training data from a selected memory region to meet the SLA for the AI ​​model ID.

[0059] In one configuration, the rack-pooled AI model memory controller 230 may receive a request from the first processor 214 to store training data in the pooled memory subsystem 200. For example, the first processor 214 may request to perform a storage operation based on desired QoS (e.g., desired bandwidth), resilience level, etc. The rack-pooled AI model memory controller 230 may receive the request and determine that it cannot process the request with the desired QoS, resilience level, etc. In other words, the rack-pooled AI model memory controller 230 may be unable to fulfill the storage request (or write request) for the first processor 214. In this case, the rack-pooled AI model memory controller 230 may send a negative acknowledgment (NACK) to the first processor 214, and may forward the request as a peer-to-peer (P2P) request to a second rack-pooled AI model memory controller (or a second memory controller) in the data center. The second rack-pooled AI model memory controller may be outside the pooled memory subsystem 200 but within the data center. The rack-pooled AI model memory controller 230 can maintain information about multiple rack-pooled AI model memory controllers in the data center and their respective capabilities. Based on this information, the rack-pooled AI model memory controller 230 can forward requests to a second rack-pooled AI model memory controller, and the second rack-pooled AI model memory controller can initiate a storage operation requested by the first processor 214.

[0060] Figure 3 A system 300 for storing training data 312 in pooled memory 330 is illustrated. System 300 may include a processor 310 operable to provide training data 312. System 300 may include a hardware platform 320 running a model 322. System 300 may include pooled memory 330, which includes a memory region 332. System 300 may include a memory controller 340. Memory controller 340 may receive a request from processor 310 for storing training data 312. The request may include training data 312 to be stored, and a model identifier (ID) identifying the model 322 associated with the training data 312. Memory controller 340 may initiate a storage operation to store the training data 312 associated with the model ID in a memory region 332 allocated for the model ID in pooled memory 330. The training data 312 stored in memory region 332 in pooled memory 330 may be addressed based on the model ID.

[0061] Another example provides a method 400 for initiating operations at the memory controller, such as Figure 4The flowchart is shown in [the diagram]. The method can be executed as instructions on a machine, wherein the instructions are included on at least one computer-readable medium or a non-transitory machine-readable storage medium. The method may include: receiving a request at a memory controller for storing training data, wherein the request includes a model identifier (ID) identifying the model associated with the training data, as shown in box 410. The method may include: sending a write request from the memory controller to store the training data associated with the model ID in a memory region of pooled memory allocated for the model ID, wherein the training data stored in the memory region of the pooled memory is addressable based on the model ID, as shown in box 420.

[0062] Figure 5 A general-purpose computing system or device 500 that can be employed in the present art is illustrated. The computing system or device 500 may include a processor 502 communicating with a memory 504. The memory 504 may include any device, combination of devices, circuitry, etc., capable of storing, accessing, organizing, and / or retrieving data. Non-limiting examples include SANs (Storage Area Networks), cloud storage networks, volatile or non-volatile RAM, phase-change memory, optical media, hard disk drive type media, and combinations thereof.

[0063] The computing system or device 500 further includes a local communication interface 506 for connections between various components of the system. For example, the local communication interface 506 may be a local data bus and / or any associated address or control bus that may be expected.

[0064] The computing system or device 500 may also include an I / O (input / output) interface 508 for controlling the system's I / O functions and for I / O connections to devices external to the computing system or device 500. It may also include a network interface 510 for network connectivity. The network interface 510 controls network communication between the system's internal and external environments. The network interface may include wired interfaces, wireless interfaces, Bluetooth interfaces, optical interfaces, etc., including suitable combinations thereof. Furthermore, the computing system 500 may additionally include a user interface 512, a display device 514, and various other components that would be beneficial to such a system.

[0065] Processor 502 can be a single processor or multiple processors, and memory 504 can be a single memory or multiple memories. Local communication interface 506 can be used as a path to facilitate communication between any single processor, multiple processors, a single memory, multiple memories, various interfaces, etc., in any useful combination.

[0066] Various technologies, or aspects or parts thereof, may take the form of program code (i.e., instructions) embodied in a tangible medium such as a CD-ROM, hard disk drive, non-transitory computer-readable storage medium, or any other machine-readable storage medium, wherein, when the program code is loaded into a machine (e.g., a computer) and executed by the machine, the machine becomes an apparatus for practicing the various technologies. Circuitry may include hardware, firmware, program code, executable code, computer instructions, and / or software. Non-transitory computer-readable storage media may be computer-readable storage media that does not include signals. In the case of executing program code on a programmable computer, a computing device may include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. Volatile and non-volatile memory and / or storage elements may be RAM, EPROM, flash memory drive, optical disk drive, magnetic hard disk drive, solid-state drive, or other media for storing electronic data. Nodes and wireless devices may also include transceiver modules, counter modules, processing modules, and / or clock modules or timer modules. One or more programs that can implement or utilize the various techniques described herein may use application programming interfaces (APIs), reusable controls, etc. Such programs may be implemented using high-level procedural or object-oriented programming languages ​​to communicate with computer systems. However, programs may be implemented using assembly language or machine language, if desired. In any case, the language may be a compiled or interpreted language and combined with hardware implementations. Exemplary systems or devices may include, but are not limited to, laptop computers, tablet computers, desktop computers, smartphones, computer terminals and servers, database storage, and other electronic products utilizing circuitry and programmable memory, such as home appliances, smart TVs, digital video disc (DVD) players, heating, ventilation, and air conditioning (HVAC) controllers, light switches, etc.

[0067] Example

[0068] The following examples relate to specific embodiments of the invention and indicate specific features, elements, or steps that may be used or otherwise combined in implementing such embodiments.

[0069] In one example, a memory controller is provided, which includes logic. The logic can receive a request at the memory controller for storing training data, wherein the request includes a model identifier (ID) identifying the model associated with the training data. The logic can send a write request from the memory controller for storing the training data associated with the model ID in a memory region allocated for the model ID in pooled memory, wherein the training data stored in the memory region in the pooled memory is addressable based on the model ID.

[0070] In one example of a memory controller, training data stored in a memory region within pooled memory can also be addressed based on the model type of the model associated with the training data.

[0071] In one example of the memory controller, the logic is also configured to: receive from the hardware platform running the model a request to read training data from a memory region in the pooled memory allocated for a model ID; send a read request to read training data from the memory region in the pooled memory, wherein the training data is read from the memory region based on the model ID or model type of the model associated with the training data; and provide the training data to the hardware platform so that the hardware platform can use the training data to train the model.

[0072] In one example of the memory controller, the logic is also configured to register a model in response to a request received from the processor, wherein the request includes a model ID associated with the model, a desired quality of service (QoS) for the model as defined in a service level agreement (SLA), a resilience level for the model, and a memory capacity for storing training data for the model, wherein the registration of the model includes allocating a memory region in pooled memory to store training data for the model ID, and the allocation of the memory region corresponds to the memory capacity included in the request.

[0073] In one example of a memory controller, the logic is also configured to provide discovery information to one or more processors or hardware platforms to enable the discovery of available training data per model ID in pooled memory.

[0074] In one example of a memory controller, the logic is also configured to forward requests received from the processor as peer-to-peer (P2P) requests to a second memory controller in the data center, wherein the P2P request is sent to the second memory controller when the request received from the processor cannot be processed at the memory controller, and the memory controller includes information identifying the existence of the second memory controller.

[0075] In one example of the memory controller, the logic is also configured to: send write requests for storing training data associated with a model ID across multiple memory regions allocated for the model ID in pooled memory, wherein the training data is distributed across multiple memory regions to satisfy the service level agreement (SLA) for the model ID; and send read requests for reading training data from one or more of the multiple memory regions, wherein the training data is read from the selected memory region to satisfy the SLA for the model ID.

[0076] In one example of the memory controller, the logic is also configured to store training data in pooled memory and read training data from pooled memory via an auxiliary memory controller connected to either the processor or the hardware platform running the model.

[0077] In one example of the memory controller, the logic is also configured to store training data in pooled memory on a per-model-ID basis and to read training data from pooled memory.

[0078] In one example of a memory controller, pooled memory is configured to dynamically increase or decrease the amount of data associated with a specific model ID.

[0079] In one example of a memory controller, the model is an artificial intelligence (AI) model.

[0080] In one example of a memory controller, the model trained using training data is an inference artificial intelligence (AI) model.

[0081] In one example of a memory controller, the memory controller is included in a storage rack in a data center.

[0082] In one example, a system for storing training data in pooled memory is provided. The system may include a processor operable to provide training data. The system may include a hardware platform for running a model. The system may include pooled memory comprising memory regions. The system may include a memory controller comprising logic. The logic may receive a request from the processor for storing training data, wherein the request includes a model identifier (ID) identifying the model associated with the training data. The logic may send a write request for storing training data associated with the model ID in a memory region allocated for the model ID in the pooled memory, wherein the training data stored in the memory region in the pooled memory is addressable based on the model ID.

[0083] In one example of the system, training data stored in a memory region within pooled memory can also be addressed based on the model type of the model associated with the training data.

[0084] In one example of the system, the model is an artificial intelligence (AI) model.

[0085] In one example of the system, the memory controller also includes logic for performing the following operations: registering a model in response to a request received from the processor, wherein the request includes a model ID associated with the model, a desired quality of service (QoS) for the model as defined in a service level agreement (SLA), a resilience level for the model, and a memory capacity for storing training data for the model, wherein the registration of the model includes allocating a memory region in pooled memory for storing training data for the model ID, and the allocation of the memory region corresponds to the memory capacity included in the request.

[0086] In one example of the system, the memory controller also includes logic for performing the following operations: providing discovery information to one or more processors or hardware platforms to enable the discovery of available training data per model ID in the pooled memory.

[0087] In one example of the system, the memory controller also includes logic for performing the following operations: forwarding requests received from the processor as peer-to-peer (P2P) requests to a second memory controller, wherein a P2P request is sent to the second memory controller when a request received from the processor cannot be processed at the memory controller, wherein the memory controller includes information identifying the existence of the second memory controller.

[0088] In one example of the system, the memory controller also includes logic for performing the following operations: sending write requests for storing training data associated with a model ID across multiple memory regions allocated for the model ID in pooled memory, wherein the training data is distributed across the multiple memory regions to satisfy the service level agreement (SLA) for the model ID; and sending read requests for reading training data from one or more of the multiple memory regions, wherein the training data is read from the selected memory region to satisfy the SLA for the model ID.

[0089] In one example, a method for initiating an operation at a memory controller is provided. The method may include receiving a request at the memory controller for storing training data, wherein the request includes a model identifier (ID) identifying the model associated with the training data. The method may also include sending a write request from the memory controller for storing the training data associated with the model ID in a memory region allocated for the model ID in pooled memory, wherein the training data stored in the memory region in the pooled memory is addressable based on the model ID.

[0090] In one example of a method for initiating an operation at a memory controller, the method further includes: receiving from a hardware platform running a model a request to read training data from a memory region allocated for a model ID in pooled memory; sending a read request to read training data from a memory region in pooled memory, wherein the training data is read from the memory region based on a model ID or model type of the model associated with the training data; and providing the training data to the hardware platform so that the hardware platform can use the training data to train a model.

[0091] In one example of a method for initiating an operation at a memory controller, the method further includes: registering a model in response to a request received from a processor, wherein the request includes a model ID associated with the model, a desired quality of service (QoS) for the model as defined in a service level agreement (SLA), a resilience level for the model, and a memory capacity for storing training data for the model, wherein registering the model includes allocating a memory region in pooled memory for storing training data for the model ID, and the allocation of the memory region corresponds to the memory capacity included in the request.

[0092] In one example of a method for initiating an operation at a memory controller, the method further includes providing discovery information to one or more processors or hardware platforms to enable the discovery of available training data per model ID in pooled memory.

[0093] In one example of a method for initiating an operation at a memory controller, the method further includes: forwarding a request received from a processor as a peer-to-peer (P2P) request to a second memory controller in a data center, wherein the P2P request is sent to the second memory controller when the request received from the processor cannot be processed at the memory controller, wherein the memory controller includes information identifying the existence of the second memory controller.

[0094] In one example of a method for initiating an operation at a memory controller, the method further includes: sending a write request for storing training data associated with a model ID across multiple memory regions allocated for a model ID in pooled memory, wherein the training data is distributed across the multiple memory regions to satisfy a service level agreement (SLA) for the model ID; and sending a read request for reading training data from one or more of the multiple memory regions, wherein the training data is read from the selected memory region to satisfy the SLA for the model ID.

[0095] In one example of a method for initiating operations at a memory controller, the method further includes storing training data in a pooled memory and reading training data from the pooled memory via an auxiliary memory controller connected to either the processor or the hardware platform running the model.

[0096] In one example of a method for initiating operations at a memory controller, the method further includes storing training data in pooled memory on a per-model-ID basis and reading training data from the pooled memory.

[0097] In one example of a method for initiating operations at the memory controller, the model is an artificial intelligence (AI) model.

[0098] While the foregoing examples illustrate the principles of embodiments of the present invention in one or more specific applications, it will be apparent to those skilled in the art that various modifications can be made to the form, use, and details of the implementation without the exercise of the inventors, without departing from the principles and concept of this disclosure.

Claims

1. A memory controller, including logic for performing the following operations: Register the model in response to a request received from the processor, wherein, The request includes a model identifier (ID) associated with the model, the expected quality of service (QoS) for the model as defined in a service level agreement (SLA), and a memory capacity for storing training data for the model, wherein registering the model includes allocating a memory region in pooled memory for storing training data for the identifier (ID), and the allocation of the memory region corresponds to the memory capacity included in the request. At the memory controller, a request for storing training data is received, wherein the request includes the model identifier (ID); and From the memory controller, a write request is sent for storing the training data associated with the model identifier (ID) in the memory region of the pooled memory, wherein the training data stored in the memory region of the pooled memory is addressable based on the model identifier (ID).

2. The memory controller according to claim 1, wherein, The training data stored in the memory region of the pooled memory can also be addressed based on the model type of the model associated with the training data.

3. The memory controller according to claim 1, wherein, The logic is also configured as follows: Receive a request from the hardware platform running the model to read the training data from the memory region allocated for the model identifier (ID) in the pooled memory; Send a read request for reading the training data from the memory region in the pooled memory, wherein the training data is read from the memory region based on the model identifier (ID) or model type of the model associated with the training data; as well as The training data is provided to the hardware platform so that the hardware platform can use the training data to train the model.

4. The memory controller according to claim 1, wherein, The logic is also configured to provide discovery information to one or more processors or hardware platforms to enable the discovery of available training data for each model identifier (ID) in the pooled memory.

5. The memory controller according to claim 1, wherein, The logic is further configured to forward the request received from the processor as a peer-to-peer (P2P) request to a second memory controller in the data center, wherein the peer-to-peer (P2P) request is sent to the second memory controller when the request received from the processor cannot be processed at the memory controller, wherein the memory controller includes information identifying the existence of the second memory controller.

6. The memory controller according to claim 1, wherein, The logic is also configured as follows: Send a write request for storing the training data associated with the model identifier (ID) across multiple memory regions allocated for the model identifier (ID) in the pooled memory, wherein the training data is distributed across the multiple memory regions to satisfy a Service Level Agreement (SLA) for the model identifier (ID); and Send a read request for reading the training data from one or more of the plurality of memory regions, wherein the training data is read from the selected memory region to satisfy the Service Level Agreement (SLA) for the model identifier (ID).

7. The memory controller according to claim 1, wherein, The logic is further configured to store the training data in the pooled memory and read the training data from the pooled memory via an auxiliary memory controller connected to one of the processors or hardware platforms running the model.

8. The memory controller according to claim 1, wherein, The logic is also configured to store the training data in the pooled memory and read the training data from the pooled memory based on each model identifier (ID).

9. The memory controller according to claim 1, wherein, The pooled memory is configured to dynamically increase or decrease the amount of data associated with a specific model identifier (ID).

10. The memory controller according to claim 1, wherein, The model in question is an artificial intelligence (AI) model.

11. The memory controller according to claim 1, wherein, The model trained using the training data is a reasoning artificial intelligence (AI) model.

12. The memory controller according to claim 1, wherein, The memory controller is included in a storage rack in the data center.

13. A system for storing training data in a pooled memory, the system comprising: A processor that can operate to provide training data; Hardware platform, and its operating model; Pooled memory, which includes memory regions; as well as A memory controller, which includes logic for performing the following operations: A model is registered in response to a request received from the processor, wherein the request includes a model identifier (ID) associated with the model, a desired quality of service (QoS) for the model as defined in a service level agreement (SLA), and a memory capacity for storing training data for the model, wherein the registration of the model includes allocating a memory region in the pooled memory for storing training data for the identifier (ID), and the allocation of the memory region corresponds to the memory capacity included in the request; The processor receives a request to store the training data, wherein the request includes the model identifier (ID); and Send a write request for storing the training data associated with the model identifier (ID) in the memory region of the pooled memory, wherein the training data stored in the memory region of the pooled memory is addressable based on the model identifier (ID).

14. The system according to claim 13, wherein, The training data stored in the memory region of the pooled memory can also be addressed based on the model type of the model associated with the training data.

15. The system according to claim 13, wherein, The model in question is an artificial intelligence (AI) model.

16. The system according to claim 13, wherein, The memory controller also includes logic for performing the following operations: providing discovery information to one or more of the processors or hardware platforms to enable the discovery of available training data for each model identifier (ID) in the pooled memory.

17. The system according to claim 13, wherein, The memory controller further includes logic for performing the following operations: forwarding the request received from the processor as a peer-to-peer (P2P) request to a second memory controller, wherein the peer-to-peer (P2P) request is sent to the second memory controller when the request received from the processor cannot be processed at the memory controller, wherein the memory controller includes information identifying the existence of the second memory controller.

18. The system according to claim 13, wherein, The memory controller also includes logic for performing the following operations: Send a write request for storing the training data associated with the model identifier (ID) across multiple memory regions allocated for the model identifier (ID) in the pooled memory, wherein the training data is distributed across the multiple memory regions to satisfy the service level agreement (SLA) for the model identifier (ID); and Send a read request for reading the training data from one or more of the plurality of memory regions, wherein the training data is read from the selected memory region to satisfy the Service Level Agreement (SLA) for the model identifier (ID).

19. A method for initiating an operation at a memory controller, the method comprising: A model is registered in response to a request received from a processor, wherein the request includes a model identifier (ID) associated with the model, a desired quality of service (QoS) for the model as defined in a service level agreement (SLA), and a memory capacity for storing training data for the model, wherein the registration of the model includes allocating a memory region in pooled memory for storing training data for the identifier (ID), and the allocation of the memory region corresponds to the memory capacity included in the request; A request for storing training data is received at the memory controller, wherein the request includes the model identifier (ID); and A write request is sent from the memory controller to store the training data associated with the model identifier (ID) in the memory region of the pooled memory, wherein the training data stored in the memory region of the pooled memory is addressable based on the model identifier (ID).

20. The method of claim 19, further comprising: Receive a request from the hardware platform running the model to read the training data from the memory region allocated for the model identifier (ID) in the pooled memory; Send a read request for reading the training data from the memory region in the pooled memory, wherein the training data is read from the memory region based on the model identifier (ID) or model type of the model associated with the training data; as well as The training data is provided to the hardware platform so that the hardware platform can use the training data to train the model.

21. The method of claim 19, further comprising: Send a write request for storing the training data associated with the model identifier (ID) across multiple memory regions allocated for the model identifier (ID) in the pooled memory, wherein the training data is distributed across the multiple memory regions to satisfy the service level agreement (SLA) for the model identifier (ID); and Send a read request for reading the training data from one or more of the plurality of memory regions, wherein the training data is read from the selected memory region to satisfy the Service Level Agreement (SLA) for the model identifier (ID).

22. The method of claim 19, further comprising: The training data is stored in the pooled memory and read from the pooled memory based on each model identifier (ID).

Images