Data caching method and device, computer device, storage medium and program product

Abstract

The application relates to a data caching method and device, computer equipment, a storage medium and a program product. A target device only needs to pull sub-cache data allocated by itself from a shared storage space, does not need to pull overall cache data corresponding to a cache task of a cache group from the shared storage space, and other nodes in the cache group synchronously acquire the sub-cache data through the target device, so that the number of concurrent access times of the nodes in the cache group to the backend shared storage space is greatly reduced, the IO pressure of the backend storage is reduced, data diffusion between nodes is completed by using a high-speed network, the overall preheating time is shortened, and the influence of a traditional preheating mode on the running of other computing jobs is avoided.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data caching method, apparatus, computer equipment, storage medium, and program product. Background Technology

[0002] During the training and inference process, artificial intelligence (AI) models process datasets containing a large number of small files, such as images and videos. Each time a small file is read, an input / output (IO) operation is performed on the file system storing the dataset. Insufficient input / output operations per second (IOPS) of the file system can lead to data read request blocking, becoming a system performance bottleneck.

[0003] In traditional technologies, computing nodes that serve as cache media are usually predetermined. Each computing node that serves as cache media pulls the data needed for AI training tasks from the file system to improve the IOPS processing capability of AI during training and inference.

[0004] However, when a large number of compute nodes concurrently pull the required data from the file system to the local storage medium for pre-warming, the backend storage system will be subjected to enormous IO pressure, affecting the execution of other services. Summary of the Invention

[0005] Therefore, it is necessary to provide a data caching method, apparatus, computer equipment, storage medium, and program product to address the aforementioned technical problems, which can reduce the I / O pressure on the storage system and avoid affecting the execution of other services.

[0006] In a first aspect, this application provides a data caching method, applied to a target device corresponding to any cache medium in a cache group, comprising:

[0007] When a sub-cached task is assigned by the master device, the sub-cached data corresponding to the sub-cached task is obtained from the shared storage space corresponding to the cache group according to the task information of the sub-cached task; wherein, the sub-cached task is obtained by the master device splitting the cache task of the cache group, and the master device is the master device in the storage device corresponding to each cache medium in the cache group;

[0008] The sub-cache data is stored in the cache medium corresponding to the target device, and the sub-cache data is synchronized to other devices; wherein, the sub-cache data is used to trigger the other devices to store the sub-cache data in the corresponding cache medium, and the other devices are devices other than the target device among the various storage devices.

[0009] In the above embodiments, the target device only needs to pull its allocated sub-cache data from the shared storage space, without needing to pull the overall cache data corresponding to the cache tasks of the cache group from the shared storage space. Furthermore, other nodes in the cache group synchronously obtain sub-cache data through the target device. Therefore, the number of concurrent accesses to the backend shared storage space by the nodes in the cache group is greatly reduced, the IO pressure of the backend storage is reduced, and the data diffusion is completed by using the high-speed network between nodes, which shortens the overall warm-up time and avoids the impact of traditional warm-up methods on the operation of other computing jobs.

[0010] In one embodiment, synchronizing the sub-cache data to other devices includes at least one of the following:

[0011] The sub-cache data is synchronized to a first device in each storage device other than the target device, and a cache data notification is sent to a second device in the first device; wherein, the number of the first devices is less than the target number; the target number is the number of storage devices corresponding to each cache medium in the cache group; the cache data notification is used to instruct the second device to synchronize the sub-cache data to a corresponding third device, wherein the third device is the storage device corresponding to a storage medium that does not store the sub-cache data.

[0012] Synchronize the sub-cache data to the third device corresponding to the target device.

[0013] In the above embodiments, on the one hand, by hierarchical synchronization, network congestion caused by the target device synchronizing to all nodes at the same time is avoided, and the network bandwidth between nodes is fully utilized to improve diffusion efficiency; on the other hand, sub-cached data is synchronized to the third device corresponding to the target device.

[0014] In one embodiment, the method further includes at least one of the following:

[0015] Based on the device identifier of the first device, select the second device from the first devices, and based on the device identifier of each third device, determine the third device corresponding to the second device;

[0016] Based on the device identifier of each third device, the third device corresponding to the target device is determined.

[0017] In the above embodiments, the second and third devices are selected based on the device identifier, which ensures the uniqueness and orderliness of device pairing and avoids data synchronization conflicts.

[0018] In one embodiment, the method further includes:

[0019] Obtain the synchronization information of the sub-cache data generated by the master device;

[0020] The synchronization information is generated based on the feedback information sent by the first device in response to the completion of the sub-cached data storage, and the synchronization information includes at least the device identifier of the first device and the device identifier of the third device.

[0021] In the above embodiments, the target device obtains the synchronization information generated by the master device based on the feedback information, and keeps abreast of the cache status of the first and third devices. It can dynamically adjust the data synchronization strategy to avoid invalid synchronization and duplicate synchronization, further improve the scheduling efficiency of the diffusion preheating, and ensure the orderly execution of cache tasks.

[0022] In one embodiment, the sub-cached data includes the complete cached data corresponding to the sub-cached task, or the data fragments corresponding to the sub-cached task.

[0023] In the above embodiments, the sub-cached data can be selected as complete data or data shards according to the data size, adapting to different dataset scenarios of small files and large files, improving the applicability of the embodiments of this application, while the data sharding method avoids the resource occupation of large file retrieval and ensures the stability of the preheating process.

[0024] In one embodiment, the sub-cache data includes data fragments of the complete cache data; obtaining the sub-cache data corresponding to the sub-cache task from the shared storage space corresponding to the cache group includes:

[0025] From the shared storage space corresponding to the cache group, obtain each data fragment corresponding to the sub-caching task in sequence;

[0026] Accordingly, storing the sub-cache data in the cache medium corresponding to the target device and synchronizing the sub-cache data to other devices includes:

[0027] Upon acquiring each data fragment, the acquired data fragment is stored in the cache medium corresponding to the target device, and the acquired data fragment is synchronized to other devices.

[0028] In the above embodiments, the method of sequentially acquiring and storing data piece by piece is adopted to achieve parallel execution of data retrieval, caching, and synchronization, which shortens the overall warm-up time, makes full use of the network and storage resources of computing nodes, and further improves the efficiency of diffusion-style warm-up.

[0029] In one embodiment, the target device is the master device among the storage devices corresponding to each cache medium in the cache group; the method further includes:

[0030] Upon obtaining a cache task, retrieve the full metadata corresponding to the cache task from the shared storage space, and determine the storage device corresponding to each cache medium in the cache group;

[0031] Based on the identified storage devices, the caching task is divided into multiple sub-caching tasks, and the metadata corresponding to each sub-caching task is determined from the full metadata.

[0032] Each sub-caching task and its corresponding metadata are sent to the corresponding storage device; the task information of each sub-caching task includes the corresponding metadata.

[0033] In the above embodiments, the master device can autonomously complete the splitting of cache tasks, metadata matching, and subtask distribution without relying on an external scheduling center. This solves the single-point bottleneck problem of centralized preheating, realizes distributed task scheduling, and improves the stability of large-scale cluster preheating.

[0034] In one embodiment, obtaining the sub-cached data corresponding to the sub-cached task from the shared storage space corresponding to the cache group based on the task information of the sub-cached task includes:

[0035] Based on the metadata corresponding to the sub-caching task, the sub-caching data corresponding to the sub-caching task is obtained from the shared storage space.

[0036] In the above embodiments, sub-cached data is accurately obtained based on the metadata corresponding to the sub-cached task, avoiding the resource consumption of blindly traversing the storage directory, improving the efficiency and accuracy of data retrieval, reducing invalid access to the shared storage space, and further alleviating the pressure on the backend storage.

[0037] In one embodiment, dividing the caching task into multiple sub-caching tasks based on the determined storage devices includes any of the following:

[0038] Obtain the load status of each storage device, and divide the cache task into multiple sub-cache tasks based on the load status of each storage device;

[0039] Based on the target number of each storage device, the caching task is evenly divided into multiple sub-caching tasks.

[0040] In the above embodiments, cache tasks can be dynamically split according to load status or evenly split according to the number of devices. This can adapt to scenarios with uneven device load as well as scenarios with consistent device configuration, improving the flexibility and rationality of task splitting, balancing the load of devices within the cache group, and ensuring the smooth operation of the preheating process.

[0041] Secondly, this application also provides a data caching device, configured in a target device corresponding to any cache medium in a cache group, comprising:

[0042] The acquisition module is used to, upon acquiring a sub-cached task allocated by the master device, acquire sub-cached data corresponding to the sub-cached task from the shared storage space corresponding to the cache group based on the task information of the sub-cached task; wherein, the sub-cached task is obtained by the master device splitting the cache task of the cache group, and the master device is the master device among the storage devices corresponding to each cache medium in the cache group;

[0043] A synchronization module is used to store the sub-cache data to the cache medium corresponding to the target device and synchronize the sub-cache data to other devices; wherein, the sub-cache data is used to trigger the other devices to store the sub-cache data to the corresponding cache medium, and the other devices are devices other than the target device among the various storage devices.

[0044] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0045] When a sub-cached task is assigned by the master device, the sub-cached data corresponding to the sub-cached task is obtained from the shared storage space corresponding to the cache group according to the task information of the sub-cached task; wherein, the sub-cached task is obtained by the master device splitting the cache task of the cache group, and the master device is the master device in the storage device corresponding to each cache medium in the cache group;

[0046] The sub-cache data is stored in the cache medium corresponding to the target device, and the sub-cache data is synchronized to other devices; wherein, the sub-cache data is used to trigger the other devices to store the sub-cache data in the corresponding cache medium, and the other devices are devices other than the target device among the various storage devices.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0048] When a sub-cached task is assigned by the master device, the sub-cached data corresponding to the sub-cached task is obtained from the shared storage space corresponding to the cache group according to the task information of the sub-cached task; wherein, the sub-cached task is obtained by the master device splitting the cache task of the cache group, and the master device is the master device in the storage device corresponding to each cache medium in the cache group;

[0049] The sub-cache data is stored in the cache medium corresponding to the target device, and the sub-cache data is synchronized to other devices; wherein, the sub-cache data is used to trigger the other devices to store the sub-cache data in the corresponding cache medium, and the other devices are devices other than the target device among the various storage devices.

[0050] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0051] When a sub-cached task is assigned by the master device, the sub-cached data corresponding to the sub-cached task is obtained from the shared storage space corresponding to the cache group according to the task information of the sub-cached task; wherein, the sub-cached task is obtained by the master device splitting the cache task of the cache group, and the master device is the master device in the storage device corresponding to each cache medium in the cache group;

[0052] The sub-cache data is stored in the cache medium corresponding to the target device, and the sub-cache data is synchronized to other devices; wherein, the sub-cache data is used to trigger the other devices to store the sub-cache data in the corresponding cache medium, and the other devices are devices other than the target device among the various storage devices.

[0053] The aforementioned data caching method, apparatus, computer equipment, storage medium, and program product allow the target device to retrieve only its allocated sub-cached data from the shared storage space, without needing to retrieve the overall cache data corresponding to the cache tasks of the cache group from the shared storage space. Furthermore, other nodes within the cache group synchronously obtain sub-cached data through the target device. This significantly reduces the number of concurrent accesses to the backend shared storage space by nodes within the cache group, lowers the IO pressure on the backend storage, and utilizes high-speed inter-node networks to complete data diffusion, shortening the overall warm-up time and avoiding the impact of traditional warm-up methods on the operation of other computing jobs. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart illustrating a data caching method in one embodiment;

[0056] Figure 2This is a schematic diagram illustrating the process of the master device issuing sub-caching tasks in one embodiment;

[0057] Figure 3 This is a data caching system architecture diagram in one embodiment;

[0058] Figure 4 This is a structural block diagram of a data caching device in one embodiment;

[0059] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] The data caching method provided in this application embodiment can be applied to application scenarios where task data for training AI models is pre-cached.

[0062] This method can be executed by any target device corresponding to any cache medium in the cache group. The target device can be a server or a terminal. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0063] The terminology used in the embodiments of this application is as follows:

[0064] A burst buffer is a buffer that temporarily stores large amounts of data to alleviate the bottleneck of traditional storage systems when processing high-speed data streams.

[0065] A cache group is a set of compute nodes that can share a local BurstBuffer cache.

[0066] Preheating involves preloading the data from the storage backend into the cache.

[0067] In one exemplary embodiment, such as Figure 1As shown, a data caching method is provided. Taking the application of this method to a target device as an example, the method includes the following steps:

[0068] S101: Upon obtaining the sub-caching task assigned by the master device, the sub-caching data corresponding to the sub-caching task is obtained from the shared storage space corresponding to the cache group based on the task information of the sub-caching task.

[0069] Among them, a cache group is a group of compute nodes that can share a local BurstBuffer cache. The management node can bind and group the Internet Protocol (IP) of multiple compute nodes according to business needs, and each group is a cache group.

[0070] The target device is any compute node in the cache group that is equipped with a BurstBuffer cache medium.

[0071] Shared storage space is a storage system, such as a distributed storage system, that provides raw datasets to be warmed up for cache groups.

[0072] The sub-caching task is obtained by the master device splitting the cache tasks of the cache group. The master device is the master device in the storage device corresponding to each cache medium in the cache group. The master device is the computing node elected within the cache group and is responsible for task splitting and task allocation.

[0073] The cache medium is one or more of the following: local SSD, NVMe, RAM disk, or distributed shared directory on the compute node, used to store BurstBuffer preheating data;

[0074] For example, after the cache group is created, the master device first sends a preheating request to the management node to obtain information on all members of the cache group. Then, based on the number of compute nodes or their load status in the cache group, it splits the overall caching task of the cache group into multiple sub-caching tasks. Each sub-caching task includes, but is not limited to, the path of the data to be cached in the shared storage space, the cache target location, and the data fragment size of the sub-caching task. The master device sends the sub-caching tasks and their corresponding metadata to the target device through the network communication module within the cache group. The target device's BurstBuffer cache management process listens to the port in real time, receives and verifies the integrity of the sub-caching tasks. After confirming that there is no data loss or information error, it marks the task as ready and waits to execute the data retrieval operation. If the target device is the master device, it can directly obtain its own sub-caching tasks without needing to obtain them through the network communication module.

[0075] The task information of the sub-caching task includes the metadata of the sub-caching data. Based on the task information of the sub-caching task, the target device locates the physical storage address of the corresponding sub-caching data in the shared storage space and pulls the sub-caching data allocated to it from the shared storage space.

[0076] In some optional implementations, the sub-cached data corresponding to the sub-cached task can be obtained from the shared storage space based on the metadata corresponding to the sub-cached task.

[0077] The metadata can include the file path, physical storage address, and data size of the sub-cached data. If the sub-cached data is a data fragment, the metadata can also include fragment information.

[0078] The target device can directly obtain the sub-cached data corresponding to the sub-cached task from the shared storage space based on the metadata corresponding to the sub-cached task.

[0079] In this way, sub-cached data is accurately retrieved based on the metadata corresponding to the sub-cached task, avoiding the resource consumption of blindly traversing the storage directory, improving the efficiency and accuracy of data retrieval, reducing invalid access to shared storage space, and further alleviating the pressure on backend storage.

[0080] S102 stores the sub-cache data in the cache medium corresponding to the target device and synchronizes the sub-cache data to other devices.

[0081] Among them, the sub-cache data is used to trigger other devices to store the sub-cache data to the corresponding cache medium. Other devices are devices other than the target device among the various storage devices.

[0082] For example, after the target device retrieves the sub-cache data, it stores the sub-cache data in the corresponding cache medium of the target device and constructs a bucket description structure to record information such as the storage location and size of the sub-cache data in the cache group. Simultaneously, it feeds back the bucket description structure of the sub-cache data to the master device. For instance, the bucket description structure includes a unique bucket identifier, the bucket size, its offset or storage path in the dataset, the node identifier of the compute node that previously cached the bucket, and a list of nodes in the cache group that have not yet retrieved the bucket.

[0083] The master device sends a notification to the target device that has cached the sub-cache data to synchronize the sub-cache data to other devices, and the target device synchronizes the sub-cache data to other devices.

[0084] If the target device is the master device, there is no need to send a notification to synchronize the sub-cache data; the sub-cache data can be directly synchronized to other devices.

[0085] For example, cache group 1 contains 3 compute nodes, with node A as the master device and nodes B and C as slave devices. The total amount of data to be preheated for the cache tasks in cache group 1 is 12GB. Master device A evenly distributes the 12GB cache task into 3 sub-cache tasks, each corresponding to 4GB of data, and allocates them to nodes A, B, and C respectively. Node B, as the target device, after obtaining the 4GB sub-cache task, pulls the 4GB of sub-cache data from the shared storage space (backend storage), stores the data in the local cache medium, and synchronizes the 4GB of sub-cache data to nodes A and C (other devices). Nodes A and C, upon receiving the data, directly store it in their local cache without accessing the backend shared storage space.

[0086] In the above embodiments, the target device only needs to pull its allocated sub-cache data from the shared storage space, without needing to pull the overall cache data corresponding to the cache tasks of the cache group from the shared storage space. Furthermore, other nodes in the cache group synchronously obtain sub-cache data through the target device. Therefore, the number of concurrent accesses to the backend shared storage space by the nodes in the cache group is greatly reduced, the IO pressure of the backend storage is reduced, and the data diffusion is completed by using the high-speed network between nodes, which shortens the overall warm-up time and avoids the impact of traditional warm-up methods on the operation of other computing jobs.

[0087] In some alternative implementations, the sub-cached data is synchronized to other devices, including at least one of the following:

[0088] For example, sub-cache data can be synchronized to a first device other than the target device in each storage device, and a cache data notification can be sent to a second device in the first device; wherein, the number of first devices is less than the target number; the target number is the number of storage devices corresponding to each cache medium in the cache group; the cache data notification is used to instruct the second device to synchronize sub-cache data to the corresponding third device, and the third device is the storage device corresponding to the storage medium that does not store sub-cache data.

[0089] Among them, the first device is a group of devices in the cache group that, excluding the target device, prioritize receiving the synchronization of sub-cache data, and its number is less than the total number of devices in the cache group; the second device is the selected device among the first devices that continues to forward data to the non-cache nodes; and the third device is a device in the cache group that does not store sub-cache data.

[0090] For example, the target device first selects a first device whose number is less than the total number of devices in the cache group, and synchronizes the sub-cache data to the first device through the group network. After the first device completes local caching, the target device sends a cache data notification to the second device in the first device. The cache data notification may carry the device identifier of the third device.

[0091] After receiving the notification, the second device synchronizes its stored sub-cache data to the corresponding third device based on the third device's device identifier, thus achieving hierarchical data diffusion. This method avoids network congestion caused by the target device simultaneously synchronizing to all nodes, fully utilizes the network bandwidth between nodes, and improves diffusion efficiency through hierarchical synchronization.

[0092] Optionally, sub-cached data can also be synchronized to a third device corresponding to the target device.

[0093] The target device directly identifies all third devices within the cache group that do not store sub-cache data, and synchronizes the sub-cache data to the third devices one by one in sequence according to the device identifier, network status, and load of the third devices.

[0094] This method has a shorter synchronization path, more direct data diffusion, and can quickly complete the caching of the entire group of data.

[0095] In some alternative implementations, when determining the second and third devices, the second device can be selected from the first devices based on the device identifier of the first device, and the third device corresponding to the second device can be determined based on the device identifier of each third device.

[0096] The target device first obtains the device identifiers of all first devices. Based on the load status of the first devices, it selects the first device with the lowest load as the second device. It then obtains the device identifiers of all third devices and assigns corresponding third devices to the second devices based on factors such as the matching degree between the device identifiers and the second device identifiers, and network distance, thereby optimizing the network transmission efficiency between the second and third devices. A second device can correspond to one or more third devices. The second device can synchronize sub-cache data to all devices (third devices) that do not store sub-cache data, or the second device and the target device can synchronize sub-cache data to all devices that do not store sub-cache data.

[0097] For example, the device identifiers of the first device are 2 and 3, and the device identifiers of the third device are 4 and 5. The target device selects node 2 (identifier 2) as the second device and assigns node 4 (identifier 4) as the corresponding third device to node 2; the target device uses node 5 (identifier 5) as its own corresponding third device, node 2 synchronizes sub-cache data to node 4, and node 1 synchronizes sub-cache data to node 5.

[0098] In this way, selecting the second and third devices based on their device identifiers ensures the uniqueness and orderliness of device pairings and avoids data synchronization conflicts.

[0099] For example, the cache group has a total of 5 devices. The target device is node 1. Nodes 2 and 3 are selected as the first devices (2 < 5), and nodes 4 and 5 are selected as the third devices. Node 1 first synchronizes the sub-cache data to nodes 2 and 3, and then sends a cache data notification to node 2 (the second device). After receiving the notification, node 2 synchronizes the sub-cache data to nodes 4 and 5. Alternatively, after receiving the notification, node 2 synchronizes the sub-cache data to node 4, and at the same time, node 1 also synchronizes the sub-cache data to node 5.

[0100] Alternatively, node 1 can first synchronize the sub-cache data to nodes 2 and 3, and then send a cache data notification to nodes 2 and 3 (the second device). After receiving the notification, node 2 synchronizes the sub-cache data to node 4, and node 3 synchronizes the sub-cache data to node 5. If node 6 also exists in the cache group, node 1 can then synchronize the sub-cache data to node 6.

[0101] Optionally, node 1 can directly synchronize sub-cached data to nodes 2, 3, 4 and 5.

[0102] In some optional implementations, the third device corresponding to the target device can be determined based on the device identifier of each third device.

[0103] For example, the device identifiers of the third device are 2, 3, 4 and 5, and nodes 2, 3, 4 and 5 are all regarded as the third device.

[0104] Similarly, the second and third devices are selected based on their device identifiers, ensuring the uniqueness and orderliness of device pairing and avoiding data synchronization conflicts.

[0105] In some optional implementations, synchronization information of the sub-cache data generated by the master device can be obtained; wherein, the synchronization information is generated based on the feedback information sent by the first device in response to the completion of sub-cache data storage, and the synchronization information includes at least the device identifier of the first device and the device identifier of the third device.

[0106] The synchronization information is generated by the master device based on the cache completion information fed back by the first device, and includes scheduling information containing the identifiers of the first and third devices. The feedback information is a task completion notification sent by the first device to the master device after completing the sub-cache data storage.

[0107] For example, after the first device completes the local caching of the sub-cached data, it immediately sends feedback information to the master device. The feedback information includes the first device identifier, the sub-cached data identifier, the caching completion time, the local cache status, etc. The master device receives and parses all the feedback information from the first device through the Burst Buffer cache management process and updates the execution status of the cache task in real time.

[0108] Based on the received feedback information, the master device filters out the first device identifier that has completed caching. Simultaneously, it combines this with the device ID of the cache group to determine the third device identifier that has not cached sub-cache data. These are then integrated to generate synchronization information. The synchronization information includes at least the first and third device identifiers, and may also include auxiliary information such as data synchronization priority and synchronization timeout. The master device can broadcast the synchronization information to the target device within the group, ensuring that the target device receives the latest device synchronization status in real time.

[0109] For example, after the first devices 2 and 3 complete the caching, they send feedback information to the master device A. The master device A generates synchronization information based on the feedback information: {first device identifier: 2, 3; third device identifier: 4, 5}, and sends it to the target device 1. After the target device 1 obtains the synchronization information, it completes the data synchronization according to the information instructions.

[0110] In the above embodiments, the target device obtains the synchronization information generated by the master device based on the feedback information, and keeps abreast of the cache status of the first and third devices. It can dynamically adjust the data synchronization strategy to avoid invalid synchronization and duplicate synchronization, further improve the scheduling efficiency of the diffusion preheating, and ensure the orderly execution of cache tasks.

[0111] In some optional implementations, the sub-cached data includes the complete cached data corresponding to the sub-cached task, or the data fragments corresponding to the sub-cached task.

[0112] Among them, complete cached data refers to all unsharded data corresponding to the sub-cached task; data shards are sub-data blocks obtained by splitting the complete cached data into fixed-size blocks.

[0113] For example, when the data volume corresponding to the sub-caching task does not exceed a preset threshold, such as less than 1GB, the main device does not perform data sharding when splitting the task, and the sub-caching data is complete, unsplit cache data. After the target device obtains the sub-caching task, it directly pulls the complete cache data from the shared storage space, completes the local caching in one go, and then synchronizes the complete cache data to other devices. This method is suitable for small file datasets, and the pull and synchronization process is simpler.

[0114] When the amount of data corresponding to a sub-cached task exceeds a preset threshold, such as 1GB, the master device will split the sub-cached data into multiple data shards of a fixed size (e.g., 1GB). When the target device fetches data, it retrieves the data one shard at a time in sequence. After caching and synchronizing each shard, it fetches the next shard, avoiding memory overflow and network congestion caused by fetching a large amount of data at once. This method is suitable for massive large file datasets and improves the stability of data processing.

[0115] In both modes, the target device records the data storage status through a bucket structure, while the master device tracks the diffusion progress of the complete data or each fragment to ensure that the full data caching is completed.

[0116] In the above embodiments, the sub-cached data can be selected as complete data or data shards according to the data size, adapting to different dataset scenarios of small files and large files, improving the applicability of the embodiments of this application, while the data sharding method avoids the resource occupation of large file retrieval and ensures the stability of the preheating process.

[0117] In some optional implementations, when the sub-cached data includes data fragments of the complete cached data, each data fragment corresponding to the sub-cached task can be obtained sequentially from the shared storage space corresponding to the cache group; after obtaining each data fragment, the obtained data fragment is stored in the cache medium corresponding to the target device, and the obtained data fragment is synchronized to other devices.

[0118] After the target device determines that the sub-cached data is a data fragment, it retrieves each data fragment sequentially from the shared storage space according to the fragmentation rules in the sub-caching task, such as fragment size, number of fragments, and fragmentation order. During the retrieval process, the target device first verifies the integrity and correctness of the fragments to avoid data corruption. Each time a fragment is retrieved, it marks it as "retrieved" and simultaneously reports the fragment retrieval progress to the master device. The master device tracks the status of each fragment through a bucket structure to ensure no fragments are missed.

[0119] Each time a target device acquires a data fragment, it immediately stores the fragment in its own cache. After local caching is complete, it directly synchronizes the fragment to other devices without waiting for other fragments. Other devices receive the fragments and immediately store them in their local caches. Once all fragments have been acquired, stored, and synchronized, they are then concatenated into a complete sub-cache data. This method achieves simultaneous fetching, caching, and synchronization, eliminating the need to wait for all fragments to be acquired before synchronization, thus shortening warm-up time and fully utilizing network and storage resources.

[0120] In the above embodiments, the method of sequentially acquiring and storing data piece by piece is adopted to achieve parallel execution of data retrieval, caching, and synchronization, which shortens the overall warm-up time, makes full use of the network and storage resources of computing nodes, and further improves the efficiency of diffusion-style warm-up.

[0121] In some optional implementations, when the target device is the master device in the storage device corresponding to each cache medium in the cache group, the master device is responsible for dividing the cache tasks of the cache group.

[0122] See Figure 2 , Figure 2A flowchart illustrating the process of a master device issuing sub-caching tasks is provided, which includes the following steps:

[0123] S201: If a cache task is obtained, retrieve the full metadata corresponding to the cache task from the shared storage space and determine the storage device corresponding to each cache medium in the cache group.

[0124] The full metadata consists of complete descriptive information about the dataset corresponding to the caching task, including file path, size, storage location, permissions, etc.

[0125] For example, after receiving a caching task, the master device retrieves the full metadata corresponding to the caching task from the shared storage space through the communication interface with the management node. The full metadata contains all descriptive information of the dataset. The master device first caches the full metadata to ensure that subsequent task splitting and data retrieval have an accurate basis. At the same time, the master device also obtains a list of all devices equipped with caching media within the cache group through the communication interface with the management node to determine the storage devices for all tasks to be assigned.

[0126] S202, based on the determined storage devices, divide the cache task into multiple sub-cache tasks, and determine the metadata corresponding to each sub-cache task from the full metadata.

[0127] The master device divides the overall caching task into multiple sub-caching tasks based on the number of storage devices, their load status, and the remaining space in the cache media within the cache group. The splitting rules can be balanced, such as distributing data evenly according to the number of devices, or load-based, such as allocating data volume according to device load. After splitting, the master device extracts the metadata corresponding to each sub-caching task from the full metadata.

[0128] S203 sends each sub-caching task and its corresponding metadata to the corresponding storage device.

[0129] Each sub-cached task includes corresponding metadata.

[0130] The master device can send each sub-cached task and its corresponding metadata to the corresponding storage device through the group communication module. After receiving the data, the storage device performs a data retrieval operation based on the metadata to ensure the accuracy of task execution.

[0131] In the above embodiments, the master device can autonomously complete the splitting of cache tasks, metadata matching, and subtask distribution without relying on an external scheduling center. This solves the single-point bottleneck problem of centralized preheating, realizes distributed task scheduling, and improves the stability of large-scale cluster preheating.

[0132] In some optional implementations, the load status of each storage device can be obtained, and the caching task can be divided into multiple sub-caching tasks according to the load status of each storage device; alternatively, the caching task can be evenly divided into multiple sub-caching tasks according to the target number of each storage device.

[0133] The master device can obtain the real-time load status of all devices through internal communication within the cache group, including CPU utilization, memory usage, remaining cache space, and network bandwidth utilization. The master device dynamically allocates data based on the load status, allocating more data to devices with low load and less data to devices with high load, ensuring load balance among devices within the cache group and preventing preheating blockage caused by overload of a single device.

[0134] Optionally, the master device can also directly split the overall cache task equally according to the total number of devices in the cache group, with each device receiving the same amount of data. This method is simple to implement and suitable for cache groups with similar device configurations and load conditions. It can quickly complete task splitting and improve scheduling efficiency.

[0135] In the above embodiments, cache tasks can be dynamically split according to load status or evenly split according to the number of devices. This can adapt to scenarios with uneven device load as well as scenarios with consistent device configuration, improving the flexibility and rationality of task splitting, balancing the load of devices within the cache group, and ensuring the smooth operation of the preheating process.

[0136] In some alternative implementations, see [link to relevant documentation]. Figure 3 , Figure 3 A data caching system architecture diagram is provided. Figure 3 The diagram shows that cache group 1 and cache group 2 include multiple compute nodes. Cache group 1 includes three compute nodes, specifically one master node and two slave nodes. The master node in each cache group interacts with the management node in the compute cluster to obtain metadata of the complete cached data corresponding to the cache task, the cache target location of the cached data, and the cache group ID. The cache group ID is used to determine the device identifier of the storage device of the cache group.

[0137] Each node (master node and slave node) in each cache group includes a cache group management module, a cache preheating management module, and a network module.

[0138] The cache group management module is used to maintain member information, node status, and task allocation rules of the cache group, while monitoring the online status and load of each node within the cache group, providing a basis for scheduling cache preheating tasks. For example, slave nodes of CacheGroup1 receive sub-caching tasks issued by the master node through this module, and nodes of CacheGroup2 join the management system of their respective cache groups through this module.

[0139] The cache preheating management module is used to receive sub-caching tasks, pull corresponding data fragments from the shared storage space, write the pulled data to the local BurstBuffer cache medium, synchronize data to other nodes in the cache group, and report the preheating progress to the cache group master node. In this embodiment, all three nodes of CacheGroup1 complete the pulling, caching, and synchronization of data fragments through this module, while the nodes of CacheGroup2 wait to receive preheating tasks or independently perform pulling operations through this module.

[0140] The network module serves as a communication channel between compute nodes and between nodes and the shared storage space. It is used for instruction interaction with the cache group master node, data retrieval communication with the shared storage space, and data synchronization transmission between nodes within the cache group. For example, the master node of CacheGroup1 issues sub-caching tasks through the network module, and slave nodes retrieve data fragments and synchronize data to other nodes in the group through the network module.

[0141] The master node also includes a cache group information module, which is used to centrally maintain all metadata of the cache group, including: the member list of the cache group (identity and network address of each member node), the load status information of each node, the remaining capacity of the cache medium, and the data shard allocation rules. In this embodiment, the master node integrates the metadata of the 12G dataset through this module, splits the dataset into three 4G sub-caching tasks (corresponding to shard 1, shard 2, and shard 3 respectively), and distributes the sub-caching tasks to each computing node in the group according to the metadata, while monitoring the warm-up progress of the entire cache group.

[0142] exist Figure 3 Based on this, the data caching method includes the following steps:

[0143] Step 1: Cache group initialization and master node election.

[0144] The management nodes are divided into CacheGroup1, which contains node 1, node 2 and node 3, based on IP range; CacheGroup2 contains node 3 and node 4.

[0145] Within CacheGroup1, node 1 can be elected as the BBGroup1 Master (ManagerNode) using the Raft Consensus Algorithm (Raft), while nodes 2 and 3 can be elected as Slave nodes.

[0146] Master node 1 maintains the list of members in the group, the network status of the nodes, and the capacity information of the cache medium through the cache group information module.

[0147] Step 2: Task splitting and metadata distribution.

[0148] Master node 1 receives the preheating task and, through the cache group management module, splits the 12G dataset into three 4G sub-cached tasks, corresponding to shard 1, shard 2, and shard 3.

[0149] Master node 1 pulls all metadata from the distributed storage, extracts the path, size, and verification information of each shard, binds it with the sub-caching task, and then distributes it to node 1, node 2, and node 3 through the network module.

[0150] Node 1 (the master node itself): Assigns shard 1.

[0151] Node 2: Assign shard 2.

[0152] Node 3: Assign shard 3.

[0153] Step 3: Nodes pull sharded data (only the target node accesses storage).

[0154] The cache preheating management modules of nodes 1, 2, and 3 receive sub-cached tasks and metadata, and only access the shared storage space to pull the corresponding shard data.

[0155] Node 1 pulls shard 1 (4G), Node 2 pulls shard 2 (4G), and Node 3 pulls shard 3 (4G).

[0156] During the fetching process, the node verifies data integrity at 1G / block intervals, and reports the progress to the master node after each block is fetched.

[0157] Node 4 belongs to CacheGroup2 and will not access storage for the time being, awaiting subsequent warm-up or independent task allocation.

[0158] Step 4: Intra-group diffusion synchronization to complete the group-wide caching.

[0159] After Node 1 pulls shard 1, it synchronizes the data of shard 1 to Node 2 and Node 3 through the network module, and writes it to the local cache medium at the same time.

[0160] After node 2 pulls shard 2, it synchronizes the data to node 1 and node 3; after node 3 pulls shard 3, it synchronizes the data to node 1 and node 2.

[0161] After all nodes receive the shards, they write them into the local BurstBuffer cache. The master node 1 verifies the cache integrity of all nodes in the group through the cache preheating management module to confirm that all shards have been successfully distributed.

[0162] Step 5: Warm-up complete, cluster enters available state.

[0163] All three nodes of CacheGroup1 have cached the complete 12G dataset, so there is no need to access the backend storage again. The training task can be started by directly reading the local cached data.

[0164] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0165] Based on the same inventive concept, this application also provides a data caching device for implementing the data caching method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more data caching device embodiments provided below can be found in the limitations of the data caching method described above, and will not be repeated here.

[0166] In one exemplary embodiment, such as Figure 4 As shown, a data caching device is provided, configured in a target device corresponding to any cache medium in a cache group, comprising:

[0167] The acquisition module 10 is used to acquire the sub-cached data corresponding to the sub-cached task from the shared storage space corresponding to the cache group based on the task information of the sub-cached task when the sub-cached task allocated by the master device is acquired; wherein, the sub-cached task is obtained by the master device splitting the cache task of the cache group, and the master device is the master device in the storage device corresponding to each cache medium in the cache group.

[0168] The synchronization module 20 is used to store the sub-cache data to the cache medium corresponding to the target device and synchronize the sub-cache data to other devices; wherein, the sub-cache data is used to trigger other devices to store the sub-cache data to the corresponding cache medium, and the other devices are devices other than the target device among the various storage devices.

[0169] In one embodiment, the synchronization module 20 is configured to perform at least one of the following:

[0170] Synchronize sub-cache data to the first device in each storage device (excluding the target device), and send cache data notification to the second device in the first device; wherein, the number of first devices is less than the target number; the target number is the number of storage devices corresponding to each cache medium in the cache group; the cache data notification is used to instruct the second device to synchronize sub-cache data to the corresponding third device, the third device being the storage device corresponding to the storage medium that does not store sub-cache data;

[0171] Synchronize sub-cache data to the third device corresponding to the target device.

[0172] In one embodiment, the device further includes a determining module for:

[0173] Based on the device identifier of the first device, select the second device from the first devices, and based on the device identifier of each third device, determine the third device corresponding to the second device; based on the device identifier of each third device, determine the third device corresponding to the target device.

[0174] In one embodiment, the acquisition module 10 is further configured to:

[0175] Obtain synchronization information of the sub-cache data generated by the master device; wherein, the synchronization information is generated based on the feedback information sent by the first device in response to the completion of sub-cache data storage, and the synchronization information includes at least the device identifier of the first device and the device identifier of the third device.

[0176] In one embodiment, the sub-cached data includes the complete cached data corresponding to the sub-cached task, or the data fragments corresponding to the sub-cached task.

[0177] In one embodiment, the sub-cached data includes data fragments of complete cached data; the acquisition module 10 is specifically used to sequentially acquire each data fragment corresponding to the sub-cached task from the shared storage space corresponding to the cache group;

[0178] The synchronization module 20 is specifically used to store the acquired data fragments in the cache medium corresponding to the target device and synchronize the acquired data fragments to other devices when each data fragment is acquired.

[0179] In one embodiment, the target device is the master device among the storage devices corresponding to each cache medium in the cache group; the apparatus further includes a sending module for:

[0180] Upon obtaining a cache task, retrieve the full metadata corresponding to the cache task from the shared storage space and determine the storage device corresponding to each cache medium in the cache group; based on the determined storage devices, divide the cache task into multiple sub-cache tasks, and determine the metadata corresponding to each sub-cache task from the full metadata; send each sub-cache task and its corresponding metadata to the corresponding storage device; wherein, the task information of each sub-cache task includes the corresponding metadata.

[0181] In one embodiment, the acquisition module 10 is specifically used for:

[0182] Based on the metadata corresponding to the sub-caching task, retrieve the sub-caching data corresponding to the sub-caching task from the shared storage space.

[0183] In one embodiment, the determining module is used to perform any of the following:

[0184] Obtain the load status of each storage device, and divide the caching task into multiple sub-caching tasks based on the load status of each storage device; divide the caching task into multiple sub-caching tasks evenly based on the target number of each storage device.

[0185] Each module in the aforementioned data caching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0186] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data caching method.

[0187] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the data caching method described in any of the above embodiments.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the data caching method described in any of the embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the data caching method described in any of the embodiments.

[0191] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0192] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0193] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0194] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data caching method, characterized in that, The method, applied to a target device corresponding to any cache medium in a cache group, includes: When a sub-cached task is assigned by the master device, the sub-cached data corresponding to the sub-cached task is obtained from the shared storage space corresponding to the cache group according to the task information of the sub-cached task; wherein, the sub-cached task is obtained by the master device splitting the cache task of the cache group, and the master device is the master device in the storage device corresponding to each cache medium in the cache group; The sub-cache data is stored in the cache medium corresponding to the target device, and the sub-cache data is synchronized to other devices; wherein, the sub-cache data is used to trigger the other devices to store the sub-cache data in the corresponding cache medium, and the other devices are devices other than the target device among the various storage devices.

2. The method according to claim 1, characterized in that, Synchronizing the sub-cache data to other devices includes at least one of the following: The sub-cache data is synchronized to a first device in each storage device other than the target device, and a cache data notification is sent to a second device in the first device; wherein, the number of the first devices is less than the target number; the target number is the number of storage devices corresponding to each cache medium in the cache group; the cache data notification is used to instruct the second device to synchronize the sub-cache data to a corresponding third device, wherein the third device is the storage device corresponding to a storage medium that does not store the sub-cache data. Synchronize the sub-cache data to the third device corresponding to the target device.

3. The method according to claim 2, characterized in that, The method further includes at least one of the following: Based on the device identifier of the first device, a second device is selected from the first devices, and based on the device identifier of each third device, the third device corresponding to the second device is determined; Based on the device identifier of each third device, the third device corresponding to the target device is determined.

4. The method according to claim 3, characterized in that, The method further includes: Obtain the synchronization information of the sub-cache data generated by the master device; The synchronization information is generated based on the feedback information sent by the first device in response to the completion of the sub-cached data storage, and the synchronization information includes at least the device identifier of the first device and the device identifier of the third device.

5. The method according to claim 1, characterized in that, The sub-cached data includes the complete cached data corresponding to the sub-cached task, or each data fragment corresponding to the sub-cached task.

6. The method according to claim 5, characterized in that, The sub-cache data includes data fragments of the complete cache data; obtaining the sub-cache data corresponding to the sub-cache task from the shared storage space corresponding to the cache group includes: From the shared storage space corresponding to the cache group, obtain each data fragment corresponding to the sub-caching task in sequence; Accordingly, storing the sub-cache data in the cache medium corresponding to the target device and synchronizing the sub-cache data to other devices includes: Upon acquiring each data fragment, the acquired data fragment is stored in the cache medium corresponding to the target device, and the acquired data fragment is synchronized to other devices.

7. The method according to any one of claims 1-4, characterized in that, The target device is the primary device among the storage devices corresponding to each cache medium in the cache group; the method further includes: Upon obtaining a cache task, retrieve the full metadata corresponding to the cache task from the shared storage space, and determine the storage device corresponding to each cache medium in the cache group; Based on the identified storage devices, the caching task is divided into multiple sub-caching tasks, and the metadata corresponding to each sub-caching task is determined from the full metadata. Each sub-caching task and its corresponding metadata are sent to the corresponding storage device; the task information of each sub-caching task includes the corresponding metadata.

8. The method according to claim 7, characterized in that, The step of retrieving sub-cached data corresponding to the sub-cached task from the shared storage space corresponding to the cache group based on the task information of the sub-cached task includes: Based on the metadata corresponding to the sub-caching task, the sub-caching data corresponding to the sub-caching task is obtained from the shared storage space.

9. The method according to claim 7, characterized in that, The step of dividing the caching task into multiple sub-caching tasks based on the determined storage devices includes any one of the following: Obtain the load status of each storage device, and divide the cache task into multiple sub-cache tasks based on the load status of each storage device; Based on the target number of each storage device, the caching task is evenly divided into multiple sub-caching tasks.

10. A data caching device, characterized in that, The device is configured as a target device corresponding to any cache medium in the cache group, the device comprising: The acquisition module is used to, upon acquiring a sub-cached task allocated by the master device, acquire sub-cached data corresponding to the sub-cached task from the shared storage space corresponding to the cache group based on the task information of the sub-cached task; wherein, the sub-cached task is obtained by the master device splitting the cache task of the cache group, and the master device is the master device among the storage devices corresponding to each cache medium in the cache group; A synchronization module is used to store the sub-cache data to the cache medium corresponding to the target device and synchronize the sub-cache data to other devices; wherein, the sub-cache data is used to trigger the other devices to store the sub-cache data to the corresponding cache medium, and the other devices are devices other than the target device among the various storage devices.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images