A cache method and device of a data set, an electronic device and a storage medium

Abstract

The application provides a data set caching method and device, electronic equipment and storage medium, and belongs to the technical field of data storage. The method comprises the following steps: determining a plurality of transition nodes from a plurality of nodes based on the data volume of a data set to be cached; determining a target node from the plurality of transition nodes; caching the data set to the target node, wherein the target node is also used to cache other data except the data set; in response to the triggering of a preset condition, detecting the current available cache space of the target node in real time; based on the current available cache space and the size of the data set, cleaning the target data in the target node until the target node caches all the data set. Through the method provided by the application, the data set caching hit rate in the cluster node is improved.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, and specifically relates to a method, apparatus, electronic device and storage medium for caching datasets. Background Technology

[0002] Users will use datasets when training scripts on an AI platform. Before being used by the training scripts, the datasets need to be cached locally or on a node.

[0003] When caching data to nodes is used, the following problems may arise: If the dataset size exceeds the node's disk space, downloading the dataset will become extremely slow when a user creates a training task and selects to use it, potentially failing due to insufficient disk space. Furthermore, the cached dataset won't be immediately cleaned up, consuming disk space and impacting other tasks. The only way to delete the dataset is for the user to manually select it from the front-end interface. However, due to insufficient disk space, the large dataset may not be fully downloaded and therefore won't be displayed in the front-end interface, making deletion impossible. If the node's disk space is already insufficient before the dataset is marked as junk data, the junk data cleanup process may fail, further consuming disk space and causing more problems. Therefore, how to cache datasets is a problem that urgently needs to be solved. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a method, apparatus, electronic device and storage medium for caching datasets, so as to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect, this application provides a method for caching a dataset, the method comprising:

[0006] For the dataset to be cached, based on the data volume of the dataset, multiple transition nodes are determined from multiple nodes; wherein the total cache space of the transition nodes is greater than the data volume;

[0007] A target node is determined from the plurality of transition nodes; wherein the available cache space of the target node is greater than or equal to the amount of data;

[0008] The dataset is cached on the target node, and the target node is also used to cache other data besides the dataset.

[0009] In response to the triggering of preset conditions, the current available cache space of the target node is detected in real time;

[0010] Based on the current available cache space and the size of the dataset, the target data in the target node is cleaned up until the target node has completely cached the entire dataset.

[0011] Optionally, before determining multiple transition nodes from multiple nodes based on the data volume of the dataset, the method includes:

[0012] Get the initial disk space capacity of each node in the cluster;

[0013] Based on the initial disk space capacity, a cache category for multiple nodes is determined; wherein, the cache category is used to characterize the size of the initial disk space capacity;

[0014] Based on the cache category, configure the total cache space for multiple nodes.

[0015] Optionally, configuring the total cache space for multiple nodes based on the cache category includes:

[0016] Determine a partitioning strategy corresponding to each of the cache categories; wherein the partitioning strategy is a strategy used to ensure that the node can function properly when caching;

[0017] According to the partitioning strategy corresponding to each node, determine the cache space capacity occupied by the node in the initial disk space;

[0018] From the initial disk space corresponding to each node, configure a total cache space with a disk space capacity equal to the cache space capacity for the node.

[0019] Optionally, determining the target data includes:

[0020] Read the file information of the cached data in the target node; wherein, the file information includes at least one of the following: file size, file caching time, and file usage frequency;

[0021] Based on the file information, the target data is determined.

[0022] Optionally, determining the target data based on the file information includes at least one of the following:

[0023] The data cached in the target node whose corresponding file size matches the size of the dataset is identified as the target data.

[0024] The data with the longest cache time in the target node is determined as the target data;

[0025] The data with the lowest file usage frequency that is cached in the target node is identified as the target data.

[0026] Optionally, when the data cached in the target node whose corresponding file size matches the size of the dataset is determined to be the target data, the method further includes:

[0027] If no data matching the size of the dataset is found, the data cached in the target node is sorted according to the file size; based on the sorting result, the target data is determined.

[0028] Optionally, the step of detecting the current available cache space of the target node in real time in response to the triggering of a preset condition includes:

[0029] Detect whether the dataset is in a pull state; wherein, the pull state is used to indicate that the target node is caching the dataset;

[0030] If so, then the current available cache space of the target node is detected in real time;

[0031] If not, then end caching of the dataset.

[0032] Optionally, if the dataset is not in a pull state, the method further includes:

[0033] Output a prompt message, which asks the user whether to enable the backup target node; wherein the backup target node is used to cache the dataset in place of the target node;

[0034] Upon receiving a confirmation operation for the prompt information, the dataset is cached using the backup target node.

[0035] A second aspect of this application provides a dataset caching device, the device comprising:

[0036] The first determining module is used to determine multiple transition nodes from multiple nodes based on the data volume of the dataset to be cached; wherein the total cache space of the transition nodes is greater than the data volume;

[0037] The second determining module is used to determine a target node from the plurality of transition nodes; wherein the available cache space of the target node is greater than or equal to the amount of data;

[0038] A caching module is used to cache the dataset to the target node, and the target node is also used to cache other data besides the dataset.

[0039] The detection module is used to detect the current available cache space of the target node in real time in response to the triggering of preset conditions;

[0040] The cleanup module is used to clean up the target data in the target node based on the current available cache space and the size of the dataset, until the target node has completely cached the entire dataset.

[0041] Optionally, the device further includes:

[0042] The configuration module is used to obtain the initial disk space capacity of each node in the cluster.

[0043] Based on the initial disk space capacity, a cache category for multiple nodes is determined; wherein, the cache category is used to characterize the size of the initial disk space capacity;

[0044] Based on the cache category, configure the total cache space for multiple nodes.

[0045] Optionally, the configuration module includes:

[0046] The third determining module is used to determine the partitioning strategy corresponding to each of the cache categories; wherein the partitioning strategy is used to ensure that the node can run normally when caching;

[0047] According to the partitioning strategy corresponding to each node, determine the cache space capacity occupied by the node in the initial disk space;

[0048] From the initial disk space corresponding to each node, configure a total cache space with a disk space capacity equal to the cache space capacity for the node.

[0049] Optionally, the cleaning module includes:

[0050] The fourth determining module is used to read the file information of the cached data in the target node; wherein, the file information includes at least one of the following: file size, file caching time, and file usage frequency;

[0051] Based on the file information, the target data is determined.

[0052] Optionally, the fourth determining module includes at least one of the following:

[0053] The first determining submodule is used to determine the target data as the data whose file size, which is cached in the target node, matches the size of the dataset.

[0054] The second determining submodule determines the data with the longest file caching time in the target node as the target data.

[0055] The third determination submodule is used to determine the data with the lowest file usage frequency that is cached in the target node as the target data.

[0056] Optionally, the first determining submodule further includes:

[0057] If no data matching the size of the dataset is found, the data cached in the target node is sorted according to the file size; based on the sorting result, the target data is determined.

[0058] Optionally, the detection module includes:

[0059] An execution module is used to detect whether the dataset is in a pull state; wherein, the pull state is used to indicate that the target node is caching the dataset;

[0060] If so, then the current available cache space of the target node is detected in real time;

[0061] If not, then end caching of the dataset.

[0062] Optionally, the execution module further includes:

[0063] The replacement module is used to output a prompt message, which asks the user whether to enable the backup target node; wherein the backup target node is used to replace the target node to cache the dataset;

[0064] Upon receiving a confirmation operation for the prompt information, the dataset is cached using the backup target node.

[0065] In a third aspect of the present invention, an electronic device is provided, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the dataset caching method as described in the first aspect of the present invention.

[0066] In a fourth aspect of the present invention, a storage medium is provided, wherein when the program is executed by a processor, the steps of the dataset caching method as described in the first aspect of the present invention are implemented.

[0067] The dataset caching method provided in this application caches the dataset to be cached. First, based on the data volume of the dataset, multiple transition nodes are determined from multiple nodes. The total cache space of the transition nodes is greater than the data volume, which can be used for the first screening to improve the cache hit rate. Then, a target node is determined from the multiple transition nodes. The available cache space of the target node is greater than or equal to the data volume. This method can further improve the cache hit rate of the dataset. Then, the dataset is cached to the target node. The target node is also used to cache other data besides the dataset. In response to the triggering of preset conditions, it is determined that the target node can be used to cache the dataset, which further improves the cache hit rate of the target node. After determining that the target node can be used to cache the dataset, the current available cache space of the target node is detected in real time. Finally, based on the current available cache space and the size of the dataset, the target data in the target node is cleaned up until the target node has completely cached the entire dataset.

[0068] The method provided by this invention has the following main advantages:

[0069] Improve the hit rate of target nodes caching the dataset. By narrowing down the range of nodes used to cache the dataset through multiple screenings, the target nodes can be selected more accurately. Furthermore, during the caching process, the available cache amount of the target nodes can be adjusted in real time until the dataset is completely cached in the target nodes, thereby improving the hit rate of target nodes caching the dataset.

[0070] This avoids issues caused by incomplete dataset caching, preventing the data from becoming unusable or causing nodes to malfunction. By adjusting the available cache size of target nodes in real time, this problem can be avoided. Furthermore, incomplete cache data can be cleaned up promptly before being marked as garbage data, thus preventing the node containing the cached dataset from becoming inoperable. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is a flowchart illustrating the steps of a dataset caching method provided in an embodiment of this application;

[0073] Figure 2 This is a flowchart illustrating a dynamic caching of a dataset provided in an embodiment of this application;

[0074] Figure 3 This is a schematic diagram of a dataset caching device provided in an embodiment of this application;

[0075] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0077] When creating tasks, if a dataset is used infrequently, it's generally recommended to cache it locally. This allows for downloading only when needed and periodically releasing the cache afterward, preventing it from consuming disk space on cluster nodes. However, if a dataset is used relatively frequently, caching it on the cluster nodes is generally recommended. Downloading a dataset each time consumes cache, degrading the AI ​​platform's performance, causing lag, and incurring time costs. Caching the dataset on cluster nodes allows for direct retrieval and use of the same dataset in subsequent task creations, improving efficiency.

[0078] However, if the dataset size exceeds the disk space of the cluster nodes, and the user creates a training task using this dataset and selects node caching, the training task will download the dataset to the specified path on the cluster nodes until disk space is insufficient. Furthermore, the cached dataset on the cluster nodes is not automatically cleaned up immediately; the user must delete it from the front-end interface. Due to insufficient disk space, this large dataset may not be fully downloaded and will not be displayed on the front-end interface. Therefore, the front-end interface cannot delete this dataset, causing the cluster nodes to be unable to download the dataset further. If this node is the master node in the cluster, the lack of cache space will cause platform service malfunctions.

[0079] It should be noted that when users cache datasets using nodes in a cluster, they typically select the dataset and the node to cache it directly in the selection interface. Once the dataset and node are selected, the node will cache the dataset.

[0080] A dataset is a collection of data that is gathered, organized, stored, and managed according to certain rules. Datasets can include various types of data, such as images, text, audio, video, sensor data, tabular data, and so on.

[0081] Example 1

[0082] The first aspect of the embodiments of this application refers to... Figure 1 , Figure 1 This is a flowchart illustrating the steps of a dataset caching method provided in an embodiment of this application. The flowchart includes:

[0083] Step S101: For the dataset to be cached, based on the amount of data in the dataset, determine multiple transition nodes from multiple nodes; wherein the total cache space of the transition nodes is greater than the amount of data.

[0084] In this embodiment, the dataset is used to input to the artificial intelligence platform and includes different types such as training dataset, prediction dataset, monitoring dataset, test dataset, meta dataset, and reference dataset. Different dataset types can perform different functions when applied to the artificial intelligence platform. Since different functions corresponding to the above datasets may be executed multiple times during the use of the artificial intelligence platform, in order to ensure the response speed and execution efficiency of the artificial intelligence, the above datasets will be cached in the cluster nodes under the artificial intelligence platform. Therefore, before using the dataset on the artificial intelligence platform, it is necessary to ensure that the dataset can be completely cached in the node and will not affect the node.

[0085] For the dataset used for caching in this invention, after the user selects the dataset, the artificial intelligence platform will first read the data volume of the dataset. Based on the data volume, multiple transition nodes will be determined from multiple nodes. Multiple nodes are all nodes included in the cluster under the same artificial intelligence platform. A node is a container that can be used to cache data. Therefore, after obtaining the data volume of the dataset, an initial screening will be performed to determine whether the node is a transition node that can meet the storage requirements of the data volume. A transition node is a node with a total cache space greater than the data volume.

[0086] Step S102: Determine the target node from the plurality of transition nodes; wherein the available cache space of the target node is greater than or equal to the amount of data.

[0087] In this embodiment, after determining the transition nodes in step S101, target nodes for caching the dataset are selected from the transition nodes. The transition nodes include target nodes. In order to ensure the hit rate of the dataset caching, the target node is a node whose available cache space is greater than or equal to the data size of the dataset. This ensures that in the initial stage of caching the dataset, the available cache space in the target node is sufficient to cache the dataset.

[0088] Step S103: Cache the dataset to the target node, and the target node is also used to cache other data besides the dataset.

[0089] In this embodiment, after the target node is determined in step S102, the target node will be used to cache the dataset. In addition to caching the dataset to be cached, the target node in this application can also cache other data, which means that the target node can cache a wide range of data and can satisfy the caching of various types of data. It is not specifically used only for caching the dataset in this application. Other data can be the same as or different from the dataset to be cached, or other data that needs to be cached under the artificial intelligence platform. This application does not make specific limitations on other data.

[0090] Step S104: In response to the triggering of preset conditions, detect the current available cache space of the target node in real time.

[0091] In this embodiment, after determining that the target node caches the dataset, it is also necessary to respond to the triggering of a preset condition. The preset condition is to determine whether the dataset is in a caching state. When caching the dataset, the dataset will first be stored in other storage spaces under the artificial intelligence platform other than the target node. After determining that the target node caches the dataset, the target node will cache the dataset. At this time, it will detect whether the state of the dataset is a caching state. When it is determined that it is a caching state, the current available cache space of the target node will be detected in real time.

[0092] Step S105: Based on the current available cache space and the size of the dataset, clean up the target data in the target node until the target node has completely cached the entire dataset.

[0093] In this embodiment, by detecting the current available cache space of the target node in real time, it can be determined how much cache space is available to cache the dataset at the current moment. This is because the process of caching the dataset is a read-processing process, and complex processing such as data augmentation and feature extraction may be performed on the dataset. Therefore, the cache space required to cache the dataset will increase accordingly. In other words, the dataset may grow larger during the caching process. Therefore, it is necessary to detect the current available cache space of the target node and the size of the dataset. The size of the dataset mentioned in this step refers to the size of the dataset that has not been cached in the target node. The size of the dataset that has not been cached in the target node can be read from the disk space according to its storage method. When the current available cache space of the target node is less than the size of the dataset, the target data that has been cached in the target node will be cleaned up until the target node has cached the entire dataset.

[0094] In one embodiment, before determining multiple transition nodes from multiple nodes based on the data volume of the dataset, the method includes: obtaining the initial disk space capacity corresponding to each of all nodes in the cluster; determining the cache category of the multiple nodes based on the initial disk space capacity; wherein the cache category is used to characterize the size of the initial disk space capacity; and configuring a total cache space for the multiple nodes based on the cache category.

[0095] In this embodiment, before determining multiple transition nodes from multiple nodes based on the data volume of the dataset, the initial disk space capacity corresponding to each node in the cluster is obtained. The cluster in this application is a cluster applied under an artificial intelligence platform, specifically a K8s cluster. Depending on the usage scenario, hardware configuration, etc., each node in a cluster will have a different initial disk space capacity. In order to ensure that the nodes run normally in the cache and improve the utilization of the node cache space, the initial disk space capacity is re-divided to determine the cache category of all nodes in the cluster. The cache category is used to characterize the size of the initial disk space. Based on the cache category, the cache space capacity is configured for multiple nodes.

[0096] In one embodiment, configuring the total cache space for multiple nodes based on the cache category includes: determining a partitioning strategy corresponding to each cache category; wherein the partitioning strategy is a strategy used to ensure that the nodes can operate normally when caching; determining the cache space capacity occupied by the node in the initial disk space according to the partitioning strategy corresponding to each node; and configuring a total cache space of disk space capacity equal to the cache space capacity for the node from the initial disk space corresponding to each node.

[0097] In this embodiment, a partitioning strategy corresponding to the cache category is determined. The role of the partitioning strategy is that when the node caches the dataset, the node generally includes functions such as managing resources, communicating with the control plane, running network proxies, and running auxiliary services. When the node runs these functions, it also needs a certain amount of space capacity to ensure that these functions can be executed normally. Therefore, this application will re-partition the initial disk space and allocate a portion of the space to ensure that the node's other functions can run normally during the process of the node caching data. There are many ways to partition the space. It can be partitioned according to the space occupied by the specific functions that the node needs to execute, or it can reduce complexity and simplify the partitioning method. As long as it can ensure that the node can execute other functions during the process of the node caching the dataset, the following will give an example of a partitioning strategy to partition the initial disk space of multiple nodes and configure the total cache space.

[0098] For example, the cache category in this application can be a large cache category and a small cache category. For instance, 200G can be used as the dividing threshold. Nodes with an initial disk space capacity greater than 200G are classified as large cache categories, and nodes with an initial disk space capacity less than or equal to 200G are classified as small cache categories.

[0099] The nodes corresponding to the large cache category will be configured with a total cache space based on a fixed space reservation. Assuming the initial disk space of the node is 400G and the fixed space reservation is 100G, then the cache space configured for this node is 300G, meaning that the node can use 300G of disk space capacity as the total cache space for this node.

[0100] The total cache space of the node corresponding to the small cache category will be configured according to the reserved ratio of the initial disk space size. Assuming that the initial disk space of the node is 100G and the reserved ratio is 20%, then the cache space configured for the node is 80G, which means that the node can use 80G of disk space capacity as the total cache space of the node.

[0101] In addition, the partitioning strategy can also set multiple initial disk space gradients for partitioning. Different gradients correspond to different fixed space reservation amounts or different reservation ratios. The specific partitioning values ​​of the node cache category are not limited in this application, as long as it can be ensured that the node is running normally during the process of caching the dataset.

[0102] In one embodiment, determining the target data includes: reading file information of cached data in the target node; wherein the file information includes at least one of file size, file caching time, and file usage frequency; and determining the target data based on the file information.

[0103] In this embodiment, the target data is data that is already cached in other data in the target node. The data cached in the node usually includes file information and file content. Different data can be distinguished by reading the file information of the cached data in the target node. The file information includes file size, file type, file caching time, access permissions, and file usage frequency, etc. This application can determine the target data by at least one of the file information of file size, file caching time, and file usage frequency.

[0104] In one embodiment, determining the target data based on the file information includes at least one of the following: determining the target data as the file size cached in the target node that matches the size of the dataset; determining the target data as the file cached in the target node with the longest cache time; and determining the target data as the file cached in the target node with the lowest usage frequency.

[0105] In this embodiment, file information cached in the target node is read. When the file size in the file information is read, data whose file size matches the size of the dataset is identified as target data. Matching means that the file size is equal to the size of the dataset. When the target node's cached dataset space is insufficient, data whose file size matches the size of the dataset will be cleaned up first. This method can ensure that the available cache space in the target node is utilized to the maximum extent.

[0106] When reading the file cache time in the file information, the data with the longest file cache time is identified as the target data. At this time, if the target node does not have enough space to cache the dataset, the data with the longest file cache time will be cleaned up first until the available cache space of the target node can cache the entire dataset.

[0107] When reading the file usage frequency in the file information, the data with the lowest file usage frequency is identified as the target data. At this time, if the target node's cached dataset space is insufficient, the data with the longest cache time in that file will be cleaned up first, until the target node's available cache space is sufficient to cache the entire dataset.

[0108] In one embodiment, when determining the data cached in the target node whose corresponding file size matches the size of the dataset as the target data, the method further includes: if no data matching the size of the dataset exists, sorting the data cached in the target node according to the file size; and determining the target data based on the sorting result.

[0109] In this embodiment, when the file size corresponding to the data cached in the target node does not match the size of the dataset, the file sizes corresponding to the data cached in the target node can be sorted from largest to smallest. Based on the sorting result, the data corresponding to that file size is cleaned up in descending order until the available cache space of the target node can cache the entire dataset.

[0110] In one embodiment, the step of detecting the current available cache space of the target node in real time in response to the triggering of a preset condition includes: detecting whether the dataset is in a pull state; wherein the pull state is used to indicate that the target node is caching the dataset; if yes, then the current available cache space of the target node is detected in real time; if no, then caching the dataset is stopped.

[0111] In this embodiment, after determining that the target node caches the dataset, it is also necessary to detect whether the dataset is in a pull state. This is because during the process of the target node caching the dataset, the dataset will be in a pull state. The pull state can indicate that the dataset is in the process of downloading or copying. If it is in this state, it proves that the target node is caching the dataset. Then, the current available cache space of the target node is detected in real time.

[0112] If the dataset is not in a pull state, it means that the target node cannot cache the dataset normally, and the process of caching the dataset on the target node will be terminated.

[0113] In one embodiment, when the dataset is not in a pull state, the method further includes: outputting a prompt message, the prompt message being used to ask the user whether to enable a backup target node; wherein the backup target node is used to replace the target node to cache the dataset; and upon receiving a confirmation operation for the prompt message, using the backup target node to cache the dataset.

[0114] In this embodiment, if the target node cannot cache the dataset, and the dataset is data that needs to be cached in the cluster, the user interface will pop up a window and output a prompt message. This prompt message asks the user whether to enable a backup target node to replace the target node for caching the dataset. The purpose is to ensure that the dataset can be cached on the cluster nodes and used on the artificial intelligence platform. Upon receiving a confirmation operation for the prompt message, the backup target node will be used to cache the dataset, ensuring that the dataset can be cached in the cluster under the artificial intelligence platform.

[0115] For example, refer to Figure 2 , Figure 2 This is a flowchart illustrating a dynamic caching of a dataset according to an embodiment of this application, used to explain the solution of this application in a specific application:

[0116] Step S201: Before the user caches data, the user will be asked to choose whether to select a node to cache the dataset. After the user selects a node to cache the dataset, the cluster will receive the instruction to cache the dataset and will enter a starting step.

[0117] Step S202: After determining that the node is caching the dataset, the task will be created. The user can select the dataset to be cached, and the source of the dataset to be cached is not limited.

[0118] Step S203: Based on the dataset determined in step S202, by reading the data size of the dataset, a suitable target node for caching the dataset will be selected. At this time, there may be multiple target nodes in the cluster nodes. Multiple target nodes can also be popped up for the user to choose from. After the user selects a target node, the artificial intelligence platform will obtain the cache path of the dataset and send it to the target node. The target node will cache the dataset according to the cache path.

[0119] Step S204: After determining that the target node caches the dataset, the scheduling status in the container where the dataset is located will be checked. The scheduling status will be used to determine whether the dataset is being cached by the target node or whether the dataset can be cached by the target node.

[0120] Step S205: Identify the scheduling status of the dataset. If it is in the pull state, it indicates that the target node can be used to cache the dataset, and proceed to step S206; if it is not in the pull state, it indicates that the target node can be used to cache the dataset, and proceed to step S210.

[0121] Step S206: During the process of the target node caching the dataset, detect the current available cache space of the target node in real time.

[0122] Step S207: After obtaining the amount of currently available cache space of the target node, determine whether the current available cache space is greater than the dataset to be cached. If yes, proceed to step S209; otherwise, proceed to step S208.

[0123] Step S208: Start data cleaning. Data cleaning cleans up the non-cached dataset data cached in the target node. The purpose of data cleaning is to ensure that the target node can cache all datasets and improve the cache hit rate of the target node. The data cleaning process can be carried out without interrupting the caching of the dataset, which can ensure the efficiency of caching. The current available cache space of the target node is dynamically updated. Steps S206 and S207 are executed until the current available cache space is greater than the dataset to be cached.

[0124] Step S209: As long as the current available cache size of the target node is greater than the dataset to be cached, the target node will continue to cache the dataset to be cached until the target node has cached the entire dataset. At this point, proceed to step S212.

[0125] Step S210: At this point, a pop-up window will appear on the caching interface, prompting the user whether to activate a backup target node to replace the target node for caching the dataset. This backup node can be one of the multiple target nodes mentioned in step S203 that was not selected by the user, or it can be any other backup node that can guarantee complete caching of the dataset. If the user enters or clicks "Yes," proceed to step S211; if the user enters or clicks "No," proceed to step S212.

[0126] Step S211: The standby target node reacquires the cache path of the dataset and caches the dataset in place of the target node until the standby target node has completely cached the entire dataset. After the cache is complete, proceed to step S212.

[0127] Step S212: End. The caching task will end after the dataset has been successfully cached, or if the target node cannot cache the dataset and the user does not choose to start a backup target node to cache the dataset. If necessary, a success or failure message can be displayed after the task ends, based on whether the dataset was successfully cached, to inform the user.

[0128] Example 2

[0129] The second aspect of the embodiments of this application refers to... Figure 3 , Figure 3 This is a schematic diagram of a dataset caching device provided in an embodiment of this application; the device includes: a first determining module 301, a second determining module 302, a caching module 303, a detection module 304, and a cleaning module 305.

[0130] The first determining module 301 is used to determine multiple transition nodes from multiple nodes based on the data volume of the dataset to be cached; wherein the total cache space of the transition nodes is greater than the data volume;

[0131] The second determining module 302 is used to determine a target node from a plurality of transition nodes; wherein the available cache space of the target node is greater than or equal to the amount of data;

[0132] Cache module 303 is used to cache the dataset to the target node, and the target node is also used to cache other data besides the dataset;

[0133] The detection module 304 is used to detect the current available cache space of the target node in real time in response to the triggering of preset conditions.

[0134] The cleaning module 305 is used to clean up the target data in the target node based on the current available cache space and the size of the dataset, until the target node has completely cached the entire dataset.

[0135] In one embodiment, the apparatus further includes: a configuration module, configured to obtain the initial disk space capacity corresponding to each of all nodes in the cluster; determine the cache category of a plurality of nodes based on the initial disk space capacity; wherein the cache category is used to characterize the size of the initial disk space capacity; and configure a total cache space for the plurality of nodes based on the cache category.

[0136] In one embodiment, the configuration module includes: a third determining module, configured to determine a partitioning strategy corresponding to each of the cache categories; wherein the partitioning strategy is a strategy to ensure that the node can operate normally when caching; determine the cache space capacity occupied by the node in the initial disk space according to the partitioning strategy corresponding to each node; and configure a total cache space of disk space capacity equal to the cache space capacity for the node from the initial disk space corresponding to each node.

[0137] In one embodiment, the cleaning module includes: a fourth determining module, configured to read file information of cached data in the target node; wherein the file information includes at least one of file size, file caching time, and file usage frequency; and determine the target data based on the file information.

[0138] In one embodiment, the fourth determining module includes at least one of the following: a first determining submodule, used to determine the data cached in the target node whose corresponding file size matches the size of the dataset as the target data; a second determining submodule, used to determine the data cached in the target node whose corresponding file has the longest cache time as the target data; and a third determining submodule, used to determine the data cached in the target node whose corresponding file has the lowest usage frequency as the target data.

[0139] In one embodiment, the first determining submodule further includes: sorting the data cached in the target node according to the file size when there is no data matching the size of the dataset; and determining the target data based on the sorting result.

[0140] In one embodiment, the detection module includes: an execution module, configured to detect whether the dataset is in a pull state; wherein the pull state is used to indicate that the target node is caching the dataset; if yes, then the current available cache space of the target node is detected in real time; if no, then caching of the dataset is terminated.

[0141] In one embodiment, the execution module further includes: a replacement module, configured to output a prompt message, the prompt message being used to ask the user whether to enable a backup target node; wherein the backup target node is used to replace the target node in caching the dataset; and upon receiving a confirmation operation for the prompt message, the backup target node is used to cache the dataset.

[0142] The dataset caching device provided by this invention, when applied to cache datasets in nodes, can not only narrow down the range of target nodes for caching datasets by filtering multiple times before the nodes cache the datasets, thus improving the hit rate of target nodes caching datasets, but also adjust the available cache amount of target nodes in real time during the caching process until the datasets are completely cached in the target nodes, further ensuring the cache hit rate of datasets. This not only solves the problem that the dataset cannot be used due to incomplete caching, but also solves the problem that the node where the cached dataset is located cannot run because the incomplete cached data is marked as garbage data and cannot be cleaned up in time.

[0143] Example 3

[0144] In a third aspect of the present invention, an electronic device is provided, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the dataset caching method as described in the first aspect of the present invention.

[0145] In this embodiment, refer to Figure 4 , Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention; as shown. Figure 4 As shown, the electronic device 100 includes a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus for communication. The memory 110 stores a computer program that can run on the processor 120 to implement the steps of the dataset caching method described in the first aspect of the present application.

[0146] Example 4

[0147] In a fourth aspect of the present invention, a storage medium is provided, wherein when the program is executed by a processor, the steps of the dataset caching method as described in the first aspect of the present invention are implemented.

[0148] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0149] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods and apparatus according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0153] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0154] The foregoing has provided a detailed description of a dataset caching method, apparatus, electronic device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for caching datasets, characterized in that, The method includes: For the dataset to be cached, based on the data volume of the dataset, multiple transition nodes are determined from multiple nodes; wherein the total cache space of the transition nodes is greater than the data volume; A target node is determined from the plurality of transition nodes; wherein the available cache space of the target node is greater than or equal to the amount of data; The dataset is cached on the target node, and the target node is also used to cache other data besides the dataset. In response to the triggering of preset conditions, the current available cache space of the target node is detected in real time; When the amount of currently available cache space is less than the size of the dataset not cached in the target node, the target data already cached in the target node is cleaned up until the target node has completely cached the entire dataset; The step of responding to the triggering of preset conditions and detecting the current available cache space of the target node in real time includes: Detect whether the dataset is in a pull state; wherein, the pull state is used to indicate that the target node is caching the dataset; If so, then the current available cache space of the target node is detected in real time; If not, then end caching of the dataset.

2. The method according to claim 1, characterized in that, Before determining multiple transition nodes from multiple nodes based on the data volume of the dataset, the method includes: Get the initial disk space capacity of each node in the cluster; Based on the initial disk space capacity, a cache category for multiple nodes is determined; wherein, the cache category is used to characterize the size of the initial disk space capacity; Based on the cache category, configure the total cache space for multiple nodes.

3. The method according to claim 2, characterized in that, The step of configuring the total cache space for multiple nodes based on the cache category includes: Determine a partitioning strategy corresponding to each of the cache categories; wherein the partitioning strategy is a strategy used to ensure that the node can function properly when caching; According to the partitioning strategy corresponding to each node, determine the cache space capacity occupied by the node in the initial disk space; From the initial disk space corresponding to each node, configure a total cache space with a disk space capacity equal to the cache space capacity for the node.

4. The method according to claim 1, characterized in that, The determination of the target data includes: Read the file information of the cached data in the target node; wherein, the file information includes at least one of the following: file size, file caching time, and file usage frequency; Based on the file information, the target data is determined.

5. The method according to claim 4, characterized in that, The determination of the target data based on the file information includes at least one of the following: The data cached in the target node whose corresponding file size matches the size of the dataset is identified as the target data. The data with the longest cache time in the target node is determined as the target data; The data with the lowest file usage frequency that is cached in the target node is identified as the target data.

6. The method according to claim 5, characterized in that, When the data cached in the target node whose corresponding file size matches the size of the dataset is determined to be the target data, the method further includes: If no data matching the size of the dataset exists, the data cached in the target node is sorted according to the file size; Based on the sorting results, the target data is determined.

7. The method according to claim 1, characterized in that, If the dataset is not in a pull state, the method further includes: Output a prompt message, which asks the user whether to enable the backup target node; wherein the backup target node is used to cache the dataset in place of the target node; Upon receiving a confirmation operation for the prompt information, the dataset is cached using the backup target node.

8. A dataset caching device, characterized in that, The device includes: The first determining module is used to determine multiple transition nodes from multiple nodes based on the data volume of the dataset to be cached; wherein the total cache space of the transition nodes is greater than the data volume; The second determining module is used to determine a target node from the plurality of transition nodes; wherein the available cache space of the target node is greater than or equal to the amount of data; A caching module is used to cache the dataset to the target node, and the target node is also used to cache other data besides the dataset. The detection module is used to detect the current available cache space of the target node in real time in response to the triggering of preset conditions; The cleanup module is used to clean up the target data already cached in the target node when the current available cache space is less than the size of the dataset not cached in the target node, until the target node has completely cached the entire dataset; The detection module includes an execution module, used to detect whether the dataset is in a pull state; wherein the pull state is used to indicate that the target node is caching the dataset; if yes, the current available cache space of the target node is detected in real time; if no, caching of the dataset is terminated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the dataset caching method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the dataset caching method as described in any one of claims 1-7.

Images