Storage acceleration system and method based on RAMdisk distributed cache and used for large model distribution deployment
By introducing a distributed cache structure based on RAMdisk in the large-model distribution deployment system and combining with the data lake storage module, the problems of slow distribution speed and high resource consumption in traditional solutions are solved, and efficient model data access and inference acceleration are achieved.
Patent Information
- Application Number
- CN202411258769.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2025-02-07
AI Technical Summary
Traditional model distribution and deployment solutions have problems such as slow distribution speed, high resource consumption, complex compatibility, low update efficiency and performance bottlenecks, especially in the deployment of large-scale deep learning models.
The storage acceleration system based on RAMdisk distributed cache is adopted, combined with the data lake storage module and distributed cache hierarchical storage structure, and RAMdisk, Memory and NVME SSD cache are used to achieve efficient hierarchical storage and fast access of data.
It significantly reduces the retrieval delay of model components, accelerates the inference process, improves overall throughput and scalability, reduces load on other storage layers, and optimizes system resource utilization.
Smart Images
Figure CN119396332A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to computing and storage system technologies, and in particular to a storage acceleration system and method based on RAMdisk distributed cache for large model distribution deployment. Background Art
[0002] The deployment of large models usually involves high-concurrency repeated reading of model files of hundreds of GB. In the past, the commonly used solutions based on image distribution, such as Figure 1 As shown in the figure. The model produced by training first needs to be stored in temporary storage to complete the image production, including data packaging, compression and other processes, and then written from the temporary storage to the persistent image warehouse. During inference deployment, it is then concurrently pulled from the image warehouse to the local storage of each inference instance, and then decompressed and data verified. It can be seen that under this scheme, the throughput mainly depends on the capacity of the underlying storage of the image warehouse, and the process requires additional overhead in both the image production and image distribution stages. At the same time, large models, especially deep learning models with millions or billions of parameters, will result in bulky images. This leads to slow distribution speeds, and downloading and uploading large images requires considerable time and bandwidth, which hinders deployment speed and scalability. And in terms of storage overhead, storing multiple versions of large images consumes a lot of storage resources. At the same time, managing different versions of models and their dependencies in the image can become complicated, leading to potential compatibility issues and challenges in rolling back to previous versions. Moreover, updating specific components of the model usually requires rebuilding the entire image, even if only a small part has changed, which is inefficient and time-consuming. Finally, each containerized model instance includes the entire runtime environment and dependencies, leading to redundancy and inefficient resource utilization, especially when deploying multiple models on the same system.
[0003] Secondly, in another acceleration solution based on traditional distributed cache, there is no memory disk, and the cache relies on slower storage layers, such as SSD or HDD, to serve all data. This inevitably leads to increased latency in accessing model components, affecting inference speed and throughput. Secondly, for performance-critical applications or models with specific latency-sensitive components, the lack of a memory disk may cause bottlenecks and limit overall system performance. And without a dedicated hot data layer, other storage layers may be overloaded due to frequent access requests, resulting in inefficient resource utilization and potential performance degradation. Finally, the inability to provide data with the lowest possible latency may hinder the scalability of the system, especially when dealing with high-volume workloads or real-time applications. .
[0004] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention
[0005] The main purpose of the present invention is to solve the problems existing in the above background technology and provide a storage acceleration system and method based on RAMdisk distributed cache for large model distribution deployment. To achieve the above purpose, the present invention adopts the following technical solutions:
[0006] In a first aspect of the present invention, a RAMdisk distributed cache based storage acceleration system for large model distribution deployment includes:
[0007] The data lake storage module is used to uniformly store model files produced by training, and has an object storage service and an event notification mechanism. The object storage service provides an interface for data lake storage and supports high-throughput data access. The event notification mechanism is used to automatically trigger notifications when data status changes, so that the cache can load or update the corresponding data.
[0008] A distributed cache hierarchical storage structure is connected to the event notification mechanism of the data lake storage module and is used for hierarchical storage according to data popularity. The distributed cache hierarchical storage structure includes:
[0009] RAMdisk cache, providing high-speed data access;
[0010] Memory cache, used to store the most popular data;
[0011] NVME SSD cache for persistent storage.
[0012] In a second aspect of the present invention, a storage acceleration method based on RAMdisk distributed cache for large model distribution deployment includes:
[0013] The model files produced by training are uniformly stored through the data lake storage module;
[0014] Use object storage services to provide data lake storage interfaces to support high-throughput data access;
[0015] When the data status changes, notifications are automatically triggered through the event notification mechanism;
[0016] Through the distributed cache hierarchical storage structure, in response to notifications from the event notification mechanism, hierarchical storage is performed according to data popularity to accelerate the reading of model data and improve system response speed;
[0017] Use RAMdisk cache to virtualize system memory into a hard disk to provide high-speed data access;
[0018] Store the most popular data in the Memory cache to ensure extremely fast reading and writing;
[0019] Store seldom-accessed data in the NVME SSD cache as persistent storage.
[0020] Furthermore, the method further comprises:
[0021] Through the automatic writing mechanism, the training output model is directly stored in the data lake;
[0022] Use event triggers to trigger event notifications when model files are written to the data lake;
[0023] Receiving notification from the event trigger via a subscriber service;
[0024] Through the preloading engine, in response to the notification received by the subscriber service, the metadata and data of the model file are preloaded into the cache close to the inference service in the distributed cache hierarchical storage structure, so that the data can be quickly read directly from the cache when the model is deployed.
[0025] The present invention has the following beneficial effects:
[0026] The present invention optimizes the model data access speed, scalability, deployment process, cost, etc. of the traditional mirror distribution solution through a data lake distributed cache solution based on object storage. In the system architecture of the present invention, the data lake storage module serves as the data source and distribution center of the distributed cache, while the distributed cache hierarchical storage structure is responsible for the fast access and caching of data. The two work together to improve the efficiency and performance of large model distribution deployment. By adding RAMdisk to the hierarchical storage architecture of the distributed cache, the advantages of RAMdisk are fully utilized to significantly reduce the delay of the retrieval model component and accelerate the reasoning process. At the same time, RAMdisk can reduce the delay so that the system can process more requests per second, thereby improving the overall throughput and scalability. It can also reduce the load on other layers and optimize the utilization of the entire system resources.
[0027] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 The traditional storage solution architecture based on image distribution in traditional model deployment distribution is shown.
[0029] Figure 2 The storage acceleration solution architecture based on RAMdisk distributed cache and data lake according to an embodiment of the present invention is shown.
[0030] Figure 3 The reading and writing process of the distributed cache according to the embodiment of the present invention is shown.
[0031] Figure 4The distributed cache hierarchical storage structure combined with RAMdisk according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0032] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.
[0033] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0034] See also Figures 2 to 4 The embodiment of the present invention provides a storage acceleration system based on RAMdisk distributed cache for large model distribution and deployment, including: a data lake storage module, which is used to uniformly store model files produced by training, and has an object storage service and an event notification mechanism, wherein the object storage service provides an interface for data lake storage and supports high-throughput data access, and the event notification mechanism is used to automatically trigger a notification when the data state changes, so that the cache performs corresponding data loading or updating; a distributed cache hierarchical storage structure, which is connected to the event notification mechanism of the data lake storage module, and is used to perform hierarchical storage according to data heat, so as to accelerate the reading of model data and improve the system response speed, and the distributed cache hierarchical storage structure includes: RAMdisk cache, which uses system memory to virtualize into a hard disk to provide high-speed data access; Memory cache, which is used to store the hottest data to ensure extremely fast reading and writing; NVME SSD cache, which is used to store less accessed data as persistent storage.
[0035] The data lake storage module and the distributed cache hierarchical storage structure work together in the system architecture of the present invention. The data lake storage module serves as the data source and distribution center of the distributed cache, while the distributed cache hierarchical storage structure is responsible for fast access and caching of data. The two work together to improve the efficiency and performance of large model distribution and deployment.
[0036] See also Figures 2 to 4The embodiment of the present invention also provides a storage acceleration method based on RAMdisk distributed cache for large model distribution and deployment, including: uniformly storing model files produced by training through a data lake storage module; using an object storage service to provide an interface for data lake storage to support high-throughput data access; automatically triggering a notification through an event notification mechanism when the data state changes; using a distributed cache hierarchical storage structure, responding to a notification from an event notification mechanism, performing hierarchical storage according to data heat to accelerate the reading of model data and improve system response speed; using RAMdisk cache to virtualize system memory into a hard disk to provide high-speed data access; storing the hottest data in the Memory cache to ensure extremely fast reading and writing; storing less accessed data in the NVME SSD cache as persistent storage.
[0037] In some embodiments, the data lake storage module specifically includes: an automatic writing mechanism for storing the training output model in the data lake; an event trigger for triggering an event notification when the model file is written to the data lake; a subscriber service for receiving notifications from the event trigger; a preloading engine for preloading the metadata and data of the model file into a cache close to the inference service in a distributed cache hierarchical storage structure in response to the notification received by the subscriber service, so that data can be quickly read directly from the cache when the model is deployed.
[0038] See also Figure 3 In some embodiments, the distributed cache hierarchical storage structure is configured to include the following read and write operation mechanisms:
[0039] For read requests: first determine whether the data of the read request already exists in the RAMdisk cache or the Memory cache; if the read request hits the cache, provide data directly from the cache; if the read request does not hit the cache, open a cache block to be used and check whether the cache block already has data; if the cache block is empty, read the required data from the NVME SSD cache as the persistence layer to the cache block, and return the read data; if the cache block already has data, write the existing data back to the persistence layer, read the required data from the persistence layer to the cache block, and return the read data.
[0040] For write requests: first determine whether the cache block corresponding to the target data of the write request already exists in the RAMdisk cache or the Memory cache; if the write request hits the cache, write the new data to the corresponding cache block; if the write request does not hit the cache, open a cache block to be used; check whether the cache block has data; if the cache block is empty, write the new data to the cache block; if the cache block already has data, write the existing data back to the persistence layer, and then write the new data to the cache block.
[0041] The specific embodiments of the present invention are further described below.
[0042] A storage acceleration solution for model distribution and deployment in large models, through a data lake distributed cache solution based on object storage, optimizes the model data access speed, scalability, deployment process, cost and other aspects of the traditional image distribution solution. At the same time, the present invention optimizes the hierarchical cache architecture of the traditional distributed cache, including technical optimization from the aspects of model deployment architecture and distributed cache architecture.
[0043] Figure 2 The storage acceleration solution architecture based on RAMdisk distributed cache and data lake of an embodiment of the present invention is shown. First of all, from the process point of view, the model produced by training is directly written into the unified data lake storage. The data lake based on object storage has the feature of event notification. When the state of an object changes, the relevant subscribers can be automatically notified in a certain way. At this time, through the event notification mechanism of the object storage, the acceleration layer can immediately perceive and preload the metadata and data of the model file into the cache close to the inference service in advance. When the model deployment is started, the data can be read directly from the cache. Secondly, from the perspective of throughput, a distributed cache based on RAMdisk, Memory and NVME SSD is designed to provide close to the inference service.
[0044] Figure 3 The read and write process of the distributed cache in the embodiment of the present invention is shown. When a user sends a request, the request type is first determined. If it is a read request, it is first determined whether the cache is hit. If it is hit, the data is directly read from the cache to complete the read operation. If the cache is not hit, a cache block to be used is opened. If there is no value in the cache block, the data is read from the persistent memory to the cache, and the cache block is confirmed to have a value. The read data is returned to complete the operation. If there is a value in the opened cache block, the value in the cache block needs to be written back to the persistent layer for storage, and then the data is read from the persistent layer to the cache block. After confirming that the cache block has a value, the read data is returned to complete the operation. ; When the request type is a write operation, if the cache is hit, the new data is directly written to the cache block, confirming that the cache block has a value, and completing the write operation. If the cache is not hit, a new cache block needs to be opened. If there is no value in the cache block, the data is read from the persistent data to the cache block, and then the new data is written to the cache block, confirming that the cache block has a value, and completing the write operation. If there is a value in the newly opened cache block, the value in the cache block needs to be stored in the persistent memory, and then the data is read from the persistent layer to the cache block, the new data is written to the cache block, confirming that the cache block has a value, and completing the write operation.
[0045] Figure 4 The distributed cache hierarchical storage structure combined with RAMdisk according to an embodiment of the present invention is shown.
[0046] The storage media in this distributed cache structure are RAMdisk, Memory and NVME SSD. Memory stores the most popular data to ensure high-speed data reading and writing; the second most popular data is stored in RAMdisk, which can also ensure high-speed reading. RAMDisk is a technology that virtualizes system memory into a hard disk drive. It can simulate a part of the memory space into a hard disk, in which users can create folders, files, etc. Since RAMDisk uses memory, its reading and writing speed is very fast, even much faster than the fastest solid-state drive now; NVME SSD stores less accessed data or large-scale components, making the memory persistent, and backing up data or using it when there is not enough memory. This distributed cache structure provides strong read throughput, and multiple instances do not need to consume a large amount of object storage I / O and bandwidth when repeatedly reading the same model data, effectively reducing the pressure on the database in a high-concurrency system and greatly improving the access speed and concurrency performance of the system.
[0047] The existing distributed cache hierarchical storage structure is optimized by combining the advantages of RAMdisk. RAMdisk provides faster access time than any other storage medium except memory. Compared with the traditional distributed cache hierarchical storage structure composed of memory, SDD and HDD, the RAMdisk storage hierarchy is obviously much faster than the read and write speed of SDD, which significantly reduces the latency of retrieving model components. By caching key model components, such as embedding layers or frequently used activation functions, RAMdisk can significantly accelerate the reasoning process, which is particularly beneficial for models with specific bottlenecks or performance-sensitive components that benefit most from ultra-low latency access, that is, predictions can be provided with minimal latency, improving the responsiveness and user experience of real-time applications (such as chatbots, translation services or online recommendations). At the same time, RAMdisk can reduce latency, enabling the system to process more requests per second, improving overall throughput and scalability. It can also reduce the load on other layers. By offloading sub-hot data to RAMdisk, the load on other layers in the distributed cache (such as NVMe SSD or HDD) is reduced. This improves the overall efficiency and responsiveness of the cache system, and lower-level storage can focus on serving less accessed data or larger model components, optimizing the overall system resource utilization.
[0048] In addition, RapidFS, a data lake storage acceleration, is added to provide more cost-effective distributed cache acceleration capabilities through near-computing deployment, support Fluid schedulers, and users can switch flexibly. Different strategies can be used to meet distribution acceleration needs of different scales. For example, for small-scale deployments with less than dozens of instances, a single copy can be used to evenly utilize the I / O capabilities of all cache disks to meet read performance requirements, while also significantly saving cache space and ensuring cache hit rates for more model data. For deployments of hundreds of instances, multiple copies can be used to improve the read throughput of the acceleration layer. When it comes to the scale of thousands of instances, performance requirements can be further met in combination with client acceleration.
[0049] In summary, the present invention proposes a new storage acceleration solution based on RAMdisk distributed cache and data lake. This structure effectively optimizes the model data access speed, scalability, deployment process, cost, etc. of the traditional mirror distribution solution. In the system architecture of the present invention, the data lake storage module serves as the data source and distribution center of the distributed cache, while the distributed cache hierarchical storage structure is responsible for the fast access and caching of data. The two work together to improve the efficiency and performance of large model distribution and deployment. The present invention optimizes the hierarchical storage architecture of the traditional distributed cache. By adding RAMdisk to the hierarchical storage architecture of the distributed cache, the advantages of RAMdisk are fully utilized to significantly reduce the delay of the retrieval model component and accelerate the reasoning process. At the same time, RAMdisk can reduce the delay so that the system can process more requests per second, thereby improving the overall throughput and scalability. It can also reduce the load on other layers and optimize the utilization of the entire system resources.
[0050] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of the reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In the absence of mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of protection of the patent application.
Claims
1. A storage acceleration system based on RAMdisk distributed cache for large model distribution deployment, characterized in that: include: The data lake storage module is used to uniformly store model files produced by training, and has an object storage service and an event notification mechanism. The object storage service provides an interface for data lake storage and supports high-throughput data access. The event notification mechanism is used to automatically trigger notifications when data status changes, so that the cache can load or update the corresponding data. A distributed cache hierarchical storage structure is connected to the event notification mechanism of the data lake storage module and is used for hierarchical storage according to data popularity. The distributed cache hierarchical storage structure includes: RAMdisk cache, providing high-speed data access; Memory cache, used to store the most popular data; NVME SSD cache for persistent storage.
2. The storage acceleration system based on RAMdisk distributed cache for large model distribution deployment as claimed in claim 1, characterized in that: The data lake storage module specifically includes: Automatic writing mechanism for storing trained models in the data lake; Event trigger, which triggers event notification when the model file is written into the data lake; A subscriber service for receiving notifications from the event trigger; The preloading engine preloads metadata and data of the model file into a cache close to the inference service in the distributed cache hierarchical storage structure in response to the notification received by the subscriber service.
3. The storage acceleration system based on RAMdisk distributed cache for large model distribution deployment as described in claim 1 or 2, characterized in that: The distributed cache hierarchical storage structure is configured to include the following read and write operation mechanisms: For read requests: First, determine whether the data requested by the read request already exists in the RAMdisk cache or the Memory cache; If the read request hits the cache, the data is provided directly from the cache; If the read request does not hit the cache, a cache block to be used is allocated and checked whether the cache block already has data; If the cache block is empty, the required data is read from the NVME SSD cache as the persistence layer to the cache block, and the read data is returned; If the cache block already has data, the existing data is written back to the persistence layer, and then the required data is read from the persistence layer to the cache block, and the read data is returned; For write requests: First, determine whether the cache block corresponding to the target data of the write request already exists in the RAMdisk cache or the Memory cache; If the write request hits the cache, the new data is written to the corresponding cache block; If the write request does not hit the cache, a cache block is allocated to be used; Check whether the cache block has data; If the cache block is empty, write the new data to the cache block; If the cache block already has data, the existing data is written back to the persistence layer, and then the new data is written to the cache block.
4. The storage acceleration system based on RAMdisk distributed cache for large model distribution deployment according to any one of claims 1 to 3, characterized in that: It also includes the use of RapidFS technology to accelerate data lake storage, provides distributed cache acceleration capabilities through near-computing deployment, and supports the Fluid scheduler.
5. A storage acceleration method based on RAMdisk distributed cache for large model distribution deployment, characterized in that: include: The model files produced by training are uniformly stored through the data lake storage module; Use object storage services to provide data lake storage interfaces to support high-throughput data access; When the data status changes, notifications are automatically triggered through the event notification mechanism; Through the distributed cache hierarchical storage structure, in response to notifications from the event notification mechanism, hierarchical storage is performed according to data popularity to accelerate the reading of model data and improve system response speed; Use RAMdisk cache to virtualize system memory into a hard disk to provide high-speed data access; Store the most popular data in the Memory cache to ensure extremely fast reading and writing; Store seldom-accessed data in the NVME SSD cache as persistent storage.
6. The storage acceleration method based on RAMdisk distributed cache for large model distribution deployment as claimed in claim 5, characterized in that: Through the automatic writing mechanism, the training output model is directly stored in the data lake; Use event triggers to trigger event notifications when model files are written to the data lake; Receiving notification from the event trigger via a subscriber service; Through the preloading engine, in response to the notification received by the subscriber service, the metadata and data of the model file are preloaded into the cache close to the inference service in the distributed cache hierarchical storage structure, so that the data can be quickly read directly from the cache when the model is deployed.
7. The storage acceleration system based on RAMdisk distributed cache for large model distribution deployment as claimed in claim 5 or 6, characterized in that: For read requests: First, determine whether the data requested by the read request already exists in the RAMdisk cache or the Memory cache; If the read request hits the cache, the data is provided directly from the cache; If the read request does not hit the cache, a cache block to be used is allocated and checked whether the cache block already has data; If the cache block is empty, the required data is read from the NVME SSD cache as the persistence layer to the cache block, and the read data is returned; If the cache block already has data, the existing data is written back to the persistence layer, and then the required data is read from the persistence layer to the cache block, and the read data is returned; For write requests: First, determine whether the cache block corresponding to the target data of the write request already exists in the RAMdisk cache or the Memory cache; If the write request hits the cache, the new data is written to the corresponding cache block; If the write request does not hit the cache, a cache block is allocated to be used; Check whether the cache block has data; If the cache block is empty, write the new data to the cache block; If the cache block already has data, the existing data is written back to the persistence layer, and then the new data is written to the cache block.
8. The storage acceleration method based on RAMdisk distributed cache for large model distribution deployment according to any one of claims 5 to 7, characterized in that: Use RapidFS technology to accelerate data lake storage, provide distributed cache acceleration capabilities through near computing deployment, and support Fluid scheduler.