Method, apparatus, electronic device and program product for caching data

Abstract

Embodiments of this disclosure relate to methods, apparatus, electronic devices, and program products for caching data. The method includes monitoring requests in a distributed file system configured with storage nodes and multiple compute nodes, wherein the compute nodes are configured with accelerator resources and storage container groups. The method further includes adding a dynamic compute node to the distributed file system in response to a request meeting predetermined conditions, and using the storage container group in the added dynamic compute node to cache data. According to embodiments of this disclosure, by configuring compute nodes, storage nodes, and dynamic compute nodes in the system, not only is the stability of the storage system guaranteed, but the system's flexibility and scalability are also improved, providing users with an efficient file storage and access experience.

Description

Technical Field

[0001] This disclosure generally relates to the field of computer technology, and more specifically to methods, apparatus, electronic devices, and program products for caching data. Background Technology

[0002] Storage systems play a crucial role in the execution of computer tasks, responsible for saving and managing all the data required for application operation. This data includes not only user-generated files, images, and videos, but also critical system files such as operating system files, applications, and hardware drivers. For example, when using container technology to execute model services, the model service needs to load large model files and continuously save checkpoint files during training.

[0003] The impact of storage systems on task execution is also reflected in data access speed, data consistency, and data reliability. Fast data access ensures efficient task execution, while data consistency and reliability ensure the accuracy and stability of task results. For example, during model inference, the model generates high-quality responses in a short time, requiring faster data access. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, electronic device, and program product for caching data.

[0005] According to a first aspect of the disclosure, a method for caching data is provided. The method includes monitoring requests in a distributed file system, wherein the distributed file system is configured with storage nodes and multiple compute nodes, and the compute nodes among the multiple compute nodes are configured with accelerator resources and storage container groups. The method also includes adding dynamic compute nodes to the distributed file system in response to requests meeting predetermined conditions. Furthermore, the method includes using the storage container groups in the added dynamic compute nodes to cache data.

[0006] In a second aspect of the disclosure, an apparatus for caching data is provided. The apparatus includes a request monitoring module configured to monitor requests in a distributed file system, wherein the distributed file system is configured with storage nodes and multiple compute nodes, and the compute nodes among the multiple compute nodes are configured with accelerator resources and storage container groups. The apparatus also includes a dynamic compute node addition module configured to add dynamic compute nodes to the distributed file system in response to requests meeting predetermined conditions. Furthermore, the apparatus includes a data caching module configured to use the storage container groups in the added dynamic compute nodes to cache data.

[0007] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to the first aspect.

[0008] In a fourth aspect of this disclosure, a computer program product is provided. The computer-readable storage medium stores computer-executable instructions, which are executed by a processor to implement the method according to the first aspect.

[0009] The summary section is intended to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment in which some embodiments of this disclosure may be implemented is shown;

[0012] Figure 2 Flowcharts of methods for caching data according to some embodiments of this disclosure are shown;

[0013] Figure 3 A schematic diagram of an architecture for caching data, representing some embodiments of this disclosure, is shown.

[0014] Figure 4 The illustration shows some embodiments of the present disclosure for Remote Direct Memory Access (RDMA) between computing nodes;

[0015] Figure 5 A schematic diagram of a process for caching data according to some embodiments of this disclosure is shown;

[0016] Figure 6A Schematic diagrams illustrating some embodiments of this disclosure for adding dynamic computing nodes are shown;

[0017] Figure 6B Schematic diagrams illustrating some embodiments of this disclosure for removing dynamic computing nodes are shown;

[0018] Figure 7 Block diagrams of apparatus for caching data according to some embodiments of the present disclosure are shown; and

[0019] Figure 8 Block diagrams of electronic devices according to some embodiments of the present disclosure are shown.

[0020] In all the accompanying figures, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0024] The frequent joining and leaving of service requests in the system places stringent demands on storage services. Traditionally, dedicated storage service clusters are used specifically for data storage. When expanding storage services, existing data needs to be migrated from old storage devices to new ones. This process often results in data loss or corruption. Furthermore, data migration can impact the availability and performance of storage services. In addition, as storage capacity increases, the difficulty of management and maintenance also increases accordingly. Another architecture eliminates the need for dedicated storage servers, utilizing compute nodes within the cluster to simultaneously act as storage services. However, this design consumes significant computing resources, including central processing units (CPUs), memory, and network interface cards (NICs). Moreover, due to GPU stability issues, this architecture may reduce the stability of the storage cluster. When GPUs or other components fail, storage nodes may unexpectedly leave the cluster, impacting the availability and reliability of the entire system.

[0025] In the embodiments of this disclosure, requests in a distributed file system are continuously monitored. This system comprises storage nodes and multiple compute nodes equipped with accelerator resources and storage container groups. When a request meets preset conditions, dynamic compute nodes are added. These newly added dynamic compute nodes utilize their storage container groups to cache data, improving the overall system efficiency and response speed. This system combines the advantages of storage nodes and compute nodes as storage services, ensuring data integrity and stability while allowing direct access to storage services on local nodes or directly to files on other nodes. This optimizes system performance, reduces data access latency, and provides users with a more efficient and smoother file storage and access experience. Furthermore, this dynamic addition of compute nodes improves the system's flexibility and scalability, adapting to constantly changing workload demands.

[0026] Figure 1 Schematic diagrams of an example environment 100 in which some embodiments of this disclosure may be implemented are shown. These schematic diagrams of the example environment 100 are for illustrative purposes only and are not intended to limit the invention. Figure 1 As shown, the example environment includes compute node 108, compute node 118, compute node 128, storage node 130, network node 140, and remote object storage 150. Compute node 108 is configured with worker container group 102, storage container group 104, and other container groups 106. Similarly, compute node 118 is configured with worker container group 112, storage container group 114, and other container groups 116, and compute node 128 is configured with worker container group 122, storage container group 124, and other container groups 126. Compute nodes can be physical computing devices or virtual computing devices. Compute nodes are also configured with accelerator resources, which can be homogeneous or heterogeneous. These accelerator resources can be graphics processing units (GPUs). GPUs can be GPUs using high-speed GPU interconnect technology, enabling direct interconnection between multiple GPUs, thereby achieving expansion of video memory and performance. These accelerator resources can be used to serve model requests; for example, they can be used for model training, inference, and other computational services.

[0027] In embodiments of this disclosure, a working container group is used to run a model service, a storage container group is used to store the data or files required by the working container group, and other container groups can be other types of container engines. These container groups can run various applications, including web servers, databases, background tasks, etc. In some embodiments, the model service can be a task such as model training, inference, or fine-tuning.

[0028] In some embodiments, compute node 108, compute node 118, and compute node 128 can access each other through RDMA technology, which is a direct memory access technology that allows the memory of one computer to directly transfer data with the memory of another computer without the intervention of the operating systems of both computers.

[0029] Storage node 130 focuses on data storage and access, providing storage services for the cluster. Remote object storage 150 also stores and backs up files needed by the compute nodes, such as machine learning model files.

[0030] Network node 140 can be a router or switch, or other component that relays and distributes data. The main function of a router is to connect different networks and forward data packets between them. A router can identify the destination address of a data packet and then determine the optimal path to send it to the destination address. Routers can also filter and block malicious network traffic, protecting network security. A switch (also known as a switching hub) primarily connects multiple computers to form a local area network (LAN), enabling these computers to communicate and share resources.

[0031] When a service request, such as a model service request, is received in the system, the compute node first searches among its fellow compute nodes for model data associated with the request. If such data is found, the compute node directly accesses the storage container group on its compute node via RDMA to retrieve the associated model data. For example, if the working container group 112 on compute node 118 is going to run a model task, compute node 118 will first access the storage container group 104 or storage container group 124 on compute node 108 or compute node 128 via RDMA to confirm whether the relevant data required by compute node 118 is available. If so, it will directly access the data via RDMA and run its own working container group 114. In some embodiments, this data may be model files, checkpoint files continuously saved during model training, or fine-tuned model files, etc.

[0032] If the relevant model data is not present on the compute node, the required associated data is retrieved from the storage node via a network node. For example, if compute node 118 does not find the required data on either compute node 108 or compute node 128, it retrieves the relevant data from storage node 130 via network node 140. This involves transferring the relevant data from the storage node to the storage container group 114 of compute node 118 via a storage protocol, allowing it to run its own working container group 112. This data associated with the model service request is stored in remote object storage, and it is pre-loaded into the storage node when a model service request is requested. In some embodiments, this protocol may be the Transmission Control Protocol (TCP).

[0033] When a worker container group on a compute node completes model service, it stores the relevant data running in that worker container group in its own storage container group. Then, during system idle time, it backs up the relevant data to other compute nodes or storage nodes. The storage or compute node then synchronizes the relevant data to remote object storage. For example, if worker container group 112 on compute node 118 completes model service, it will store the completed data in storage container group 114. Storage container group 114 will back up this data to other compute nodes, such as compute node 108 or compute node 128, and storage node 130, at appropriate times. The storage and compute nodes will then synchronize the relevant data to remote object storage 150.

[0034] When there are more model service requests, and the current compute nodes in the cluster cannot meet the model service requests (named tidal requests, which exhibit periodic peaks over time), the system will dynamically create more dynamic compute nodes. Figure 1 (Not shown). This compute node is also configured with work container groups, storage container groups, and other container groups.

[0035] After a dynamic compute node is added, it will first access the compute nodes in the cluster via RDMA to obtain the data it needs. If the data is available, it will directly access the data via RDMA to run it in its own working container group. If not, it will obtain the required data from storage node 130 via the network node using the storage protocol and cache it in its own storage container group to run the model service.

[0036] After a tidal request ends, the system first checks if the compute nodes in the system already have the relevant data for the dynamic compute node. If so, the dynamic compute node is removed directly. If not, the relevant data for the dynamic node is cached on other compute or storage nodes in the system before the dynamic node is deleted. Then, during idle periods, the system will synchronize the data to remote object storage.

[0037] In the embodiments of this disclosure, requests in a distributed file system are continuously monitored. This system comprises storage nodes and multiple compute nodes equipped with accelerator resources and storage container groups. When a request meets preset conditions, dynamic compute nodes are added. These newly added dynamic compute nodes utilize their storage container groups to cache data, improving the overall system efficiency and response speed. This system combines the advantages of storage nodes and compute nodes as storage services, ensuring data integrity and stability while allowing direct access to storage services on local nodes or remote direct memory access to files on other nodes. This optimizes system performance, reduces data access latency, and provides users with a more efficient and smoother file storage and access experience. Furthermore, this dynamic addition of compute nodes improves the system's flexibility and scalability, adapting to ever-changing workload demands.

[0038] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to other environments with different structures and / or functionalities.

[0039] The following will combine Figures 2 to 8 The process according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0040] Figure 2 A flowchart of a method 200 for caching data, according to some embodiments of this disclosure, is shown. In block 202, requests in a distributed file system are monitored, wherein the distributed file system is configured with storage nodes and multiple compute nodes, and the compute nodes among the multiple compute nodes are configured with accelerator resources and storage container groups. See, for example, [reference needed]. Figure 1 The system is configured with storage nodes 130 and compute nodes 108, 118, and 128. Each compute node is configured with at least a worker container group and a storage container group. The compute nodes are also configured with accelerator resources to run computing and other services. Figure 1(Not shown). Compute nodes can directly access each other via RDMA. For example, compute nodes 108, 118, and 128 can access each other via RDMA technology without caching data in the local node's buffer. When a compute node retrieves data from a storage node, it does so via a storage protocol, and this data is replicated to the compute node's storage container group. When a service request occurs in the system, data related to the service request is loaded into the compute node's storage container group to support the compute node's work. For example, when compute node 128 executes a service request, data related to that request is loaded into storage container group 124. For example, during model inference, checkpoint files continuously saved during the process are also saved to the compute node's storage container group 124. Data files saved on the compute nodes are also synchronized to remote object storage 150. When a service request occurs in the system, data related to the service request is saved to storage node 130, and this data is then loaded into the compute node's storage container group to support the compute node's work. For example, when compute node 128 executes a service request, the data related to that request will be loaded from storage node 130 into storage container group 124. Similarly, during model inference, checkpoint files continuously saved throughout the process will also be saved to storage container group 124 of compute node 128. Data files saved on the compute node will also be synchronized to remote object storage 150.

[0041] In box 204, in response to the request meeting predetermined conditions, a dynamic computing node is added to the distributed file system. For example, when a tidal wave of requests arrives—that is, when all computing nodes in the system are busy serving existing requests—a new service request arrives. At this time, the system adds a dynamic computing node, which is also configured with a working container group and a storage container group. This dynamic computing node is configured with accelerator resources, and the added dynamic computing node is used to run the new service request. For example, if computing nodes 108, 118, and 128 in the system are all in a service state, and a new service request is detected in the system, the system will add a new dynamic computing node 160 to serve the new request. Data is loaded into the working container group of dynamic computing node 160 to run the service request.

[0042] In box 206, the storage container group in the added dynamic compute node is used to cache data. For example, when a dynamic compute node is added, the working container group of that dynamic compute node needs data to run, and this data is stored in the storage container group of that dynamic compute node, or the data generated during the run and the data after the run is completed are also stored in this storage container group. For example, checkpoint files that are continuously saved during model inference will be saved to the storage container group. For example, model files after model fine-tuning will also be saved to this storage container group.

[0043] In the embodiments of this disclosure, requests in a distributed file system are continuously monitored. This system comprises storage nodes and multiple compute nodes equipped with accelerator resources and storage container groups. When a request meets preset conditions, dynamic compute nodes are added. These newly added dynamic compute nodes utilize their storage container groups to cache data, improving the overall system efficiency and response speed. This system combines the advantages of storage nodes and compute nodes as storage services, ensuring data stability and integrity while allowing direct access to storage services on local nodes or remote direct memory access to files on other nodes. This optimizes system performance, reduces data access latency, and provides users with a more efficient and smoother file storage and access experience. Furthermore, this dynamic addition of compute nodes improves the system's flexibility and scalability, adapting to constantly changing workload demands.

[0044] Figure 3 A schematic diagram of an architecture 300 for caching data, representing some embodiments of this disclosure, is shown. (See reference...) Figure 3 The architecture includes compute nodes 308, 318, and 328, storage nodes 330, network nodes 340, remote object storage 350, and dynamic compute nodes 360. Each compute node is configured with worker container groups, storage container groups, and other container groups. Compute nodes are also configured with accelerator resources, which can be homogeneous or heterogeneous. For example, compute node 308 is configured with worker container group 302, storage container group 304, and other container groups 306. Dynamic compute nodes are the same type of node as compute nodes, and are also configured with worker container groups, storage container groups, and other container groups, as well as accelerator resources.

[0045] The working container group is used to run the model service, and the storage container group is used to store the data or files required by the working container group. Other container groups 306, 316, and 326 can be other types of container engines. Compute nodes 308, 318, and 328 can access each other via RDMA.

[0046] Storage nodes 330 focus on data storage and access, typically providing consistent and predictable performance. Remote object storage 350 stores files needed by the compute nodes, such as machine learning model files. Remote object storage 350 also backs up the data required by the entire system. Network nodes 340 can be components such as routers or switches, capable of relaying and distributing data.

[0047] refer to Figure 3 When the system is undersaturated, if a model service request is received, the compute node will first search among its fellow compute nodes for model data associated with the request. If such data is found, the compute node will directly access the storage container group on its compute node via RDMA to obtain the associated model data. For example, if the working container group 312 on compute node 318 will run a model task, compute node 318 will first access the storage container group 304 or storage container group 324 on compute node 308 or compute node 328 via RDMA to confirm whether the relevant data required by compute node 318 is available. If so, it will directly access the data via RDMA and run its own working container group 314. In some embodiments, this data may be model files, checkpoint files, or fine-tuned model files, etc.

[0048] refer to Figure 3 If the compute node does not have the relevant model data, it retrieves the necessary related data from the storage node via a network node. For example, if compute node 318 does not find the required related data on compute nodes 308 and 328, it retrieves the relevant data from storage node 330 via network node 340. This involves transferring the relevant data from the storage node to the storage container group 314 of compute node 318 via a storage protocol, allowing it to run its own working container group 312. This data associated with the model service request is stored in remote object storage, and it is pre-loaded onto the storage node when a model service request is requested. In some embodiments, this protocol can be TCP.

[0049] refer to Figure 3When a worker container group on a compute node completes model service, it stores the relevant data running in that worker container group in its own storage container group. Then, during system idle time, it backs up the relevant data to other compute nodes or storage nodes. The storage or compute node then synchronizes the relevant data to remote object storage. For example, if worker container group 322 on compute node 328 completes model service, it will store the completed data in storage container group 324. Storage container group 324 will back up this data to other compute nodes, such as compute node 308 or compute node 318, and storage node 330, at appropriate times. The storage and compute nodes will then synchronize the relevant data to remote object storage 350.

[0050] This approach combines the advantages of storage nodes and compute nodes as storage services, ensuring data stability and integrity while allowing direct access to storage services on local nodes or access to files on other nodes via remote direct memory access, thus optimizing system performance.

[0051] Continue to refer to Figure 3 When a tidal demand occurs—that is, when there are more model service requests than the current compute nodes in the cluster can meet—the system dynamically creates more dynamic compute nodes. These compute nodes are also configured with worker container groups, storage container groups, and other container groups.

[0052] Continue to refer to Figure 3 After dynamic compute node 360 ​​is added, it will first access the compute nodes in the cluster via RDMA to obtain the data it needs. If the data is available, it will directly access the data via RDMA to run it in its own working container group. If not, it will obtain the required data from storage node 330 through the network node using the storage protocol and cache it in its own storage container group to run the model service.

[0053] Continue to refer to Figure 3After a tidal request ends, the system first checks if the compute nodes in the system already have the relevant data for the dynamic compute node. If so, the dynamic compute node is removed directly. If not, the relevant data for the dynamic compute node is cached on other compute nodes or storage nodes in the system before the dynamic compute node is deleted. Then, during idle periods, the system synchronizes the data to remote object storage. For example, when dynamic compute node 360 ​​completes the model service involving tidal requests, the system checks if compute nodes 308, 318, and 328 in the cluster have the relevant data for dynamic compute node 360. If so, the system removes the dynamic compute node directly. If not, the system caches the data for the dynamic compute node on compute nodes 308, 318, and 328 via RDMA, or transfers the data from the dynamic compute node to storage node 330 via a storage protocol, and then synchronizes it to remote object storage 320. The selection of these compute nodes can consider factors such as the distance between the dynamic compute nodes and other compute nodes, or the transmission rate.

[0054] This system combines the advantages of storage nodes and compute nodes as storage services, ensuring data stability and integrity while allowing direct access to storage services on local nodes or remote direct memory access to files on other nodes. This optimizes system performance, reduces data access latency, and provides users with a more efficient and smoother file storage and access experience. Furthermore, this dynamic addition of compute nodes allows the system to flexibly respond to various changing workload demands, ensuring that the system always remains in optimal operating condition.

[0055] Figure 4 Schematic diagrams illustrating some embodiments of this disclosure for remote direct memory access between computing nodes are shown. References Figure 3 Through RDMA technology, compute nodes 308, 318, 328, and dynamic compute node 360 ​​can achieve remote direct memory access without caching data on the nodes in the local node's cache. (Reference) Figure 4 Compute node 404 is configured with storage container group 401, working container group 402, and working container group 403. Similarly, compute node 414 is also configured with storage container group 411, working container group 412, and working container group 413. The working container group is used to run model files, where services such as model training, inference, or fine-tuning can be performed. The storage container group is used to store related running files, which can be data before execution, data generated during execution, or data saved after execution. The number of working container groups and storage container groups is unlimited.

[0056] The compute nodes are also equipped with accelerator resources, which can be heterogeneous or homogeneous. These accelerator resources are used to run tasks related to service requests. Dynamic compute nodes are also equipped with accelerator resources. For example, they may be equipped with embedded neural network processing units (NPUs) specifically designed to accelerate neural network operations, addressing the inefficiency of traditional chips in neural network computations.

[0057] For example, it may be equipped with a GPU. This GPU could utilize high-speed GPU interconnect technology, enabling direct interconnection between multiple GPUs. This allows for expansion of video memory and performance to meet the demands of the largest visual computing workloads, and this technology makes it ideal for handling massively parallel computing tasks. Another example is the configuration of a high-performance hardware accelerator, whose architecture allows the accelerator to perform general matrix multiplication and other operations in parallel, thereby accelerating deep learning workflows. Furthermore, such accelerator devices have large amounts of memory, providing space to accommodate very large models.

[0058] The compute nodes are also equipped with Remote Network Interface Cards (RNICs). For example, compute node 404 is equipped with RNIC 405-1 and RNIC 405-2. Similarly, compute node 414 is equipped with RNIC 415-1 and RNIC 415-2. Communication between compute nodes uses RDMA technology, i.e., they are connected via RDMA-enabled RNICs. For example, compute node 404 and compute node 414 are connected via RNIC 405-2 and RNIC 415-2. Dynamic compute nodes are also configured with RNICs that support RDMA.

[0059] In RDMA communication, applications can directly access the memory of a remote computer through the RNIC without copying data to the operating system's buffer. For example, compute node 404 can directly access the memory of compute node 414 without copying relevant data from compute node 414 to its own compute node 404's buffer. Another example is... (See reference...) Figure 3 If the data needed by dynamic computing node 360 ​​resides on computing node 328, and both dynamic computing node 360 ​​and computing node 328 are configured with RNICs supporting RDMA, then dynamic computing node 360 ​​can directly access the relevant data on computing node 328 without needing to copy this data to its own node 360. However, if the data needed by dynamic computing node 360 ​​resides on storage node 330, then dynamic computing node 360 ​​needs to transfer the required data from storage node 330 to its own node 360 ​​via a storage protocol.

[0060] By configuring RNICs that support RDMA on compute nodes and dynamic compute nodes, the number of data copies and context switching overhead can be reduced, thereby improving communication performance and efficiency and bringing users a better service experience.

[0061] Figure 5 Schematic diagrams illustrating processes for caching data according to some embodiments of this disclosure are shown. References Figure 5 In box 502, the system continuously monitors tidal requests. These tidal requests refer to service requests that the current compute nodes in the cluster cannot fulfill, requiring additional compute nodes to meet the incoming requests. These requests can be model training requests, model inference requests, or model fine-tuning requests, among other computational requests. In box 504, once a tidal request is identified, a dynamic compute node will be added to the system, as shown in box 506. After adding the dynamic compute node, the request will be served; specifically, in box 508, the container groups on the added dynamic compute nodes will run according to the tidal request. These runs can include performing services such as model inference, training, or fine-tuning, or executing background tasks.

[0062] In box 510, it is determined whether the tidal request has been completed. If no tidal request has arrived, the container cluster continues to operate normally; that is, the compute nodes in the system continue to run and store relevant runtime data in the storage container group or storage node on the compute node. If the tidal request has been completed, meaning that the system's original computing power can meet the service requests in the service cluster, then in box 512, the relevant runtime data of the added dynamic compute node will be cached on the existing compute or storage nodes in the system. In box 516, the added dynamic compute node will be deleted. Once the data on the added dynamic compute node has been completely cached on the system's compute or storage nodes, the added dynamic compute node can be deleted.

[0063] For example, if a group of working containers on a dynamic compute node has completed model services, and the data from these model services is stored in a storage container group on that dynamic compute node, when the working container group completes its task, the system will determine whether other compute nodes' storage container groups or storage nodes already contain the relevant data generated by the dynamic compute node. If so, the dynamic storage node will be directly deleted. If not, the relevant data needs to be backed up to the compute or storage nodes in the system. Specifically, this data will be transferred directly to the compute node via RDMA, or to the storage node via a storage protocol. The compute and storage nodes in the system will then synchronize this data to the remote object storage. At this point, in box 514, the container cluster continues to run, and the container cluster continues to run normal service requests, synchronizing the relevant data generated during the process to the remote object storage.

[0064] This scalable method of dynamically adding compute nodes not only increases the computing power the system urgently needs but also caches the data generated by the added compute nodes. This precise data caching mechanism greatly optimizes system performance and significantly reduces data access latency. This strategy allows the system to dynamically allocate compute nodes according to actual needs, ensuring full utilization of hardware resources and thus significantly reducing hardware storage costs.

[0065] Figure 6A Schematic diagrams illustrating some embodiments of this disclosure for adding dynamic computing nodes are shown. References Figure 6A Dynamic compute nodes are added to the 602A. These dynamic compute nodes are configured with worker container groups, storage container groups, and other container groups serving background tasks. Accelerator resources are configured on these added dynamic compute nodes. These dynamic compute nodes belong to the same type as the compute nodes in the system. These dynamic compute nodes, compute nodes, and dynamic compute nodes communicating with each other can achieve mutual communication via RDMA, i.e., direct access to data on these nodes. These compute nodes, dynamic compute nodes, and storage nodes in the system communicate with each other via storage protocols.

[0066] In version 604A, the system checks if the added dynamic compute node has the necessary data. If so, in version 606A, the added dynamic compute node can directly access the relevant data on the compute node via RDMA technology, without needing to copy it to the dynamic compute node's storage container group or buffer. (See reference...) Figure 3 If the data needed by the dynamic computing node 360 ​​is on the computing node 308, then the relevant data can be accessed directly through RDMA.

[0067] In section 604A, it checks if the existing compute nodes in the system have the data required by the newly added dynamic compute node. If not, in section 608A, the added dynamic compute node can obtain the necessary data through its storage protocol with the storage nodes; that is, it needs to copy the data to the dynamic compute node's storage container group or buffer. For example, see [reference needed]. Figure 3 If the data required by the dynamic computing node 360 ​​resides on the storage node 330, the relevant data can be transferred to the dynamic computing node via TCP and loaded into the storage container group for use by the working container group. This data can be at least a portion of a machine learning model file, such as checkpoint files or model files.

[0068] This strategy of adding dynamic computing nodes allows the system to dynamically allocate computing nodes according to actual needs, ensuring full utilization of hardware resources. Furthermore, through a precise data caching mechanism, system performance is greatly optimized and data access latency is significantly reduced. In addition, this strategy eliminates the need for additional storage nodes, thus significantly reducing hardware costs.

[0069] Figure 6B Schematic diagrams illustrating some embodiments of this disclosure for removing dynamic computing nodes are shown. References Figure 6B In 602B, the tidal request has been completed. The system's original computing power is sufficient to meet normal service requests, and no additional dynamic computing nodes are needed. The computing nodes in the system can also complete the incoming service requests normally.

[0070] In 604B, it determines whether the existing compute or storage nodes in the system store data related to the operation of the newly added dynamic compute node. For example, refer to... Figure 3 The system checks whether the data on the newly added dynamic node 360 ​​is backed up on compute nodes 308, 318, 328, and storage node 330. If a backup exists, the newly added dynamic compute node is deleted directly in step 606B. If no backup exists, the system transfers the relevant data from the dynamic compute node to the system's compute nodes via RDMA or to the system's storage nodes via a storage protocol in step 608B.

[0071] For example, if the system does not find relevant data, such as checkpoint files, related to the operation of dynamic compute node 360 ​​on compute nodes 308, 318, 328, and storage node 330, it will cache the relevant data of dynamic compute node 360 ​​on compute nodes 308, 318, or 328 via RDMA. Alternatively, it can cache the relevant data of dynamic compute node 360 ​​on storage node 330 via a storage protocol. This data will also be synchronized to remote object storage to ensure data integrity and stability.

[0072] In this way, the data processed by the added dynamic computing nodes will not be at risk of being lost. Furthermore, this two-level caching mechanism provides users with a smoother and more efficient file storage and access experience.

[0073] Figure 7 A block diagram of an apparatus 700 for caching data, according to some embodiments of the present disclosure, is shown. Figure 7 As shown, the device 700 includes a request monitoring module 702, configured to monitor requests in a distributed file system. The distributed file system is configured with storage nodes and multiple compute nodes, and the compute nodes are configured with accelerator resources and storage container groups. The device 700 also includes a dynamic compute node addition module 704, configured to add dynamic compute nodes to the distributed file system in response to requests meeting predetermined conditions. The device 700 also includes a data caching module 706, configured to use the storage container groups in the added dynamic compute nodes to cache data.

[0074] Figure 8 Block diagrams of electronic devices 800 according to some embodiments of the present disclosure are shown. Device 800 may be the device or apparatus described in the embodiments of the present disclosure. Figure 8 As shown, device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of device 800. The CPU / GPU 801, ROM 802, and RAM 803 are interconnected via bus 808. Input / output (I / O) interface 805 is also connected to bus 804. Although not shown in... Figure 8 As shown, device 800 may also include a coprocessor.

[0075] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0076] The various methods or processes described above can be executed by CPU / GPU 801. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU / GPU 801, one or more steps or actions in the methods or processes described above can be performed.

[0077] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0078] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0079] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0080] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0081] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0082] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0084] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0085] The following are some example implementations of this disclosure.

[0086] Example 1. A method for caching data, comprising:

[0087] Monitor requests in a distributed file system, which is configured with storage nodes and multiple compute nodes, and the compute nodes among the multiple compute nodes are configured with accelerator resources and storage container groups.

[0088] In response to the request meeting predetermined conditions, a dynamic computing node is added to the distributed file system; and

[0089] The data is cached using the storage container group in the added dynamic computing node.

[0090] Example 2. The method described in Example 1 further includes:

[0091] Determine the data associated with the request;

[0092] In response to the computing node storing data associated with the request, the data is accessed through the computing node; and

[0093] In response to the fact that the computing node does not store the data associated with the request, the data in the storage node is transferred to the dynamic computing node.

[0094] Example 3. The method according to any one of Examples 1-2 further includes:

[0095] Running a group of work containers on the dynamic computing node; and

[0096] The data is loaded into the working container group of the dynamic computing node, and the data includes at least a portion of the machine learning model file.

[0097] Example 4. The method according to any one of Examples 1-3, wherein using the storage container group in the added dynamic computing node to cache data includes:

[0098] The storage container group in the dynamic computing node is used to cache the data associated with the request; and

[0099] The storage container group in the dynamic compute node is used to cache the data generated by the working container group in the dynamic compute node.

[0100] Example 5. The method according to any one of Examples 1-4 further includes:

[0101] In response to the completion of the request, the dynamic computing node is removed.

[0102] Example 6. The method according to any one of Examples 1-5, wherein removing the dynamic computing node includes:

[0103] In response to the fact that at least one of the storage node and the computing node stores the data in the dynamic computing node, the dynamic computing node is removed;

[0104] In response to the fact that neither the storage node nor the compute node stores the data of the dynamic compute node:

[0105] Transmit the data from the dynamic computing node to the computing node; or

[0106] Transmit the data from the dynamic computing node to the storage node; and

[0107] Remove the dynamic computing node.

[0108] Example 7. The method according to any one of Examples 1-6, wherein transmitting data from the dynamic computing node to the computing node comprises:

[0109] The data is loaded from the dynamic computing node to the computing node via remote direct content access, without going through the dynamic computing node and the processor of the computing node.

[0110] Example 8. The method according to any one of Examples 1-7, wherein transmitting the data from the dynamic computing node to the storage node comprises:

[0111] The data is loaded from the dynamic computing node to the storage node via a storage protocol based on the transmission control protocol.

[0112] Example 9. The method according to any one of Examples 1-8 further includes:

[0113] Synchronize the data of the storage node to a remote object storage service; and

[0114] The data in the computing node is synchronized to the remote object storage service.

[0115] Example 10. An apparatus for caching data, comprising:

[0116] The request monitoring module is configured to monitor requests in a distributed file system, which is configured with storage nodes and multiple computing nodes, and the computing nodes among the multiple computing nodes are configured with accelerator resources and storage container groups.

[0117] A dynamic computing node addition module is configured to add a dynamic computing node to the distributed file system in response to the request meeting predetermined conditions; and

[0118] The data caching module is configured to use the storage container group in the added dynamic computing node to cache data.

[0119] Example 11. The apparatus according to Example 10 further includes:

[0120] The data determination module is configured to determine the data associated with the request;

[0121] A data access module is configured to access data through the computing node in response to the computing node storing data associated with the request; and

[0122] The data transmission module is configured to, in response to the computing node not storing data associated with the request, transfer the data from the storage node to the dynamic computing node.

[0123] Example 12. The apparatus according to any one of Examples 10-11 further includes:

[0124] The work container group runtime module is configured to run work container groups on the dynamic compute node; and

[0125] A first data loading module is configured to load the data into the working container group of the dynamic computing node, the data including at least a portion of a machine learning model file.

[0126] Example 13. The apparatus according to any one of Examples 10-12, wherein the data buffer module comprises:

[0127] The associated data caching module is configured to use a group of storage containers in the dynamic computing node to cache data associated with the request; and

[0128] A data caching module is configured to use the storage container group in the dynamic computing node to cache data generated by the working container group in the dynamic computing node.

[0129] Example 14. The apparatus according to any one of Examples 10-13 further includes:

[0130] The dynamic computing node removal module is configured to remove the dynamic computing node in response to the completion of the request.

[0131] Example 15. The apparatus according to any one of Examples 10-14, wherein the dynamic computing node removal module comprises:

[0132] The first dynamic node removal module is configured to remove the dynamic computing node in response to at least one of the storage node and the computing node storing data in the dynamic computing node.

[0133] The second dynamic node removal module is configured to respond when neither the storage node nor the compute node stores data for the dynamic compute node:

[0134] The first data transmission module is configured to transmit data from the dynamic computing node to the computing node; or

[0135] The second data transmission module is configured to transmit data from the dynamic computing node to the storage node; and

[0136] The third dynamic node removal module is configured to remove the dynamic computing node.

[0137] Example 16. The apparatus according to any one of Examples 10-15, wherein the first data transmission module comprises:

[0138] The second data loading module is configured to load the data from the dynamic computing node to the computing node via remote direct memory access, without going through the dynamic computing node and the processor of the computing node.

[0139] Example 17. The apparatus according to any one of Examples 10-16, wherein the second data transmission module comprises:

[0140] The data loading module is configured to load the data from the dynamic computing node to the storage node via a storage protocol based on the transmission control protocol.

[0141] Example 18. The apparatus according to any one of Examples 10-17 further includes:

[0142] The first data synchronization module is configured to synchronize the data of the storage node to a remote object storage service; and

[0143] The second data synchronization module is configured to synchronize the data in the computing node to the remote object storage service.

[0144] Example 19. An electronic device comprising:

[0145] Processor; and

[0146] A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform actions, the actions including:

[0147] Monitor requests in a distributed file system, which is configured with storage nodes and multiple compute nodes, and the compute nodes among the multiple compute nodes are configured with accelerator resources and storage container groups.

[0148] In response to the request meeting predetermined conditions, a dynamic computing node is added to the distributed file system; and

[0149] The data is cached using the storage container group in the added dynamic computing node.

[0150] Example 20. The electronic device according to Example 19, further comprising:

[0151] Determine the data associated with the request;

[0152] In response to the computing node storing data associated with the request, the data is accessed through the computing node; and

[0153] In response to the fact that the computing node does not store the data associated with the request, the data in the storage node is transferred to the dynamic computing node.

[0154] Example 21. The electronic device according to any one of Examples 19-20, wherein the operation further includes:

[0155] Running a group of work containers on the dynamic computing node; and

[0156] The data is loaded into the working container group of the dynamic computing node, and the data includes at least a portion of the machine learning model file.

[0157] Example 22. An electronic device according to any one of Examples 19-21, wherein using the storage container group in the added dynamic computing node to cache data includes:

[0158] The storage container group in the dynamic computing node is used to cache the data associated with the request; and

[0159] The storage container group in the dynamic compute node is used to cache the data generated by the working container group in the dynamic compute node.

[0160] Example 23. The electronic device according to any one of Examples 19-22, further comprising:

[0161] In response to the completion of the request, the dynamic computing node is removed.

[0162] Example 24. An electronic device according to any one of Examples 19-23, wherein removing the dynamic computing node comprises:

[0163] In response to the fact that at least one of the storage node and the computing node stores the data in the dynamic computing node, the dynamic computing node is removed;

[0164] In response to the fact that neither the storage node nor the compute node stores the data of the dynamic compute node:

[0165] Transmit the data from the dynamic computing node to the computing node; or

[0166] Transmit the data from the dynamic computing node to the storage node; and

[0167] Remove the dynamic computing node.

[0168] Example 25. An electronic device according to any one of Examples 19-24, wherein transmitting data from the dynamic computing node to the computing node comprises:

[0169] The data is loaded from the dynamic computing node to the computing node via remote direct content access, without going through the dynamic computing node and the processor of the computing node.

[0170] Example 26. An electronic device according to any one of Examples 19-25, wherein transmitting data from the dynamic computing node to the storage node comprises:

[0171] The data is loaded from the dynamic computing node to the storage node via a storage protocol based on the transmission control protocol.

[0172] Example 27. The electronic device according to any one of Examples 19-26, further comprising:

[0173] Synchronize the data of the storage node to a remote object storage service; and

[0174] The data in the computing node is synchronized to the remote object storage service.

[0175] Example 28. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of Examples 1 to 9.

[0176] Example 29. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 9.

[0177] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for caching data, comprising: Monitoring requests in a distributed file system, which is configured with storage nodes and multiple compute nodes, wherein the compute nodes are configured with accelerator resources and storage container groups, and the compute nodes run on worker container groups for executing the requests; In response to the request meeting predetermined conditions, a dynamic computing node is added to the distributed file system; Data is cached using the storage container group in the added dynamic compute node, wherein the cached data includes data generated by the working container group in the dynamic compute node; and In response to the completion of the request, the dynamic computing node is removed; The removal of the dynamic computing node includes: In response to the fact that at least one of the storage node and the computing node stores the data in the dynamic computing node, the dynamic computing node is removed; The method further includes: Determine the data associated with the request; In response to the computing node storing data associated with the request, the data is accessed through the computing node; and In response to the fact that the computing node does not store the data associated with the request, the data in the storage node is transferred to the dynamic computing node.

2. The method according to claim 1, further comprising: The group of work containers runs on the dynamic computing node; as well as The data is loaded into the working container group of the dynamic computing node, and the data includes at least a portion of the machine learning model file.

3. The method of claim 1, wherein using the storage container group in the added dynamic computing node to cache data comprises: The storage container group in the dynamic computing node is used to cache the data associated with the request.

4. The method according to claim 1, wherein removing the dynamic computing node further comprises: In response to the fact that neither the storage node nor the compute node stores the data of the dynamic compute node: Transmit the data from the dynamic computing node to the computing node; or The data from the dynamic computing node is transferred to the storage node; as well as Remove the dynamic computing node.

5. The method according to claim 4, wherein transmitting the data from the dynamic computing node to the computing node comprises: The data is loaded from the dynamic computing node to the computing node via remote direct memory access, without going through the dynamic computing node and the processor of the computing node.

6. The method of claim 4, wherein transmitting the data from the dynamic computing node to the storage node comprises: The data is loaded from the dynamic computing node to the storage node via a storage protocol based on the transmission control protocol.

7. The method according to claim 1, further comprising: Synchronize the data of the storage node to the remote object storage service; as well as The data in the computing node is synchronized to the remote object storage service.

8. An apparatus for caching data, comprising: The request monitoring module is configured to monitor requests in a distributed file system, which is configured with storage nodes and multiple computing nodes. The computing nodes are configured with accelerator resources and storage container groups. The computing nodes run on a group of working containers, which are used to execute the requests. A dynamic computing node addition module is configured to add a dynamic computing node to the distributed file system in response to the request meeting predetermined conditions. A data caching module is configured to cache data using a group of storage containers in the added dynamic computing node, wherein the cached data includes data generated by the group of working containers in the dynamic computing node; The removal module is configured to remove the dynamic computing node in response to the completion of the request; The removal module includes: A dynamic computing node removal module is configured to remove the dynamic computing node in response to at least one of the storage node and the computing node storing data in the dynamic computing node. The device further includes: The data determination module is configured to determine the data associated with the request; An access module is configured to access data through the computing node in response to the computing node storing data associated with the request; and A transmission module is configured to transfer the data from the storage node to the dynamic computing node in response to the computing node not storing the data associated with the request.

9. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 7.

10. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 7.

Images