Preloading method and device of training data, computer device and storage medium

Abstract

The present disclosure relates to a preloading method and device of training data, a computer device and a storage medium. The method comprises: obtaining node information of a plurality of available cache nodes and training data of a model training task in a target cluster when it is detected that a current cache node fails; determining cache priorities of the plurality of available cache nodes for loading the training data according to the node information of the plurality of available cache nodes respectively; determining at least one candidate available cache node from the plurality of available cache nodes according to the training data; determining a target available cache node from the at least one candidate available cache node according to the cache priorities, and controlling the target available cache node to preload the training data. In this way, the fault tolerance, resource utilization and business stability of the target cluster can be significantly improved in a large-scale training or other scenarios with high requirements for data timeliness.

Description

Technical Field

[0001] This disclosure relates to the field of model training, and more particularly to a method, apparatus, computer device, and storage medium for preloading training data. Background Technology

[0002] In practical applications, some data in remote storage can be cached in local cache nodes through manual pre-configuration, thereby improving data read and write speed.

[0003] In related technologies, model training is typically time-consuming, during which cache nodes may fail. To ensure the continuous operation of model training tasks, the training task is usually rescheduled to other cache nodes to resume training after a cache node fails. However, cache node failures are unpredictable, and the newly assigned cache node often fails to load the required training data in advance. Therefore, when the cache node first reads the training data, it still faces problems such as high access latency and slow reading speed, affecting the recovery efficiency of the training task. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, computer device, and storage medium for preloading training data to solve the problems existing in the related art.

[0005] A first aspect of this disclosure provides a method for preloading training data. The method includes: when a failure of a current cache node is detected, acquiring node information of multiple available cache nodes in a target cluster and training data included in the current cache node; determining the cache priority of loading training data for the multiple available cache nodes based on the node information of the multiple available cache nodes; determining at least one candidate available cache node from the multiple available cache nodes based on the training data; determining a target available cache node from the at least one candidate available cache node based on the cache priority, and controlling the target available cache node to preload training data.

[0006] A second aspect of this disclosure provides a training data preloading apparatus applied to the training data preloading selection method of the first aspect. The apparatus includes: an acquisition module, configured to acquire node information of multiple available cache nodes in a target cluster and training data included in the current cache node when a failure of the current cache node is detected; a first determination module, configured to determine the cache priority of loading training data by the multiple available cache nodes based on the node information of each available cache node; a second determination module, configured to determine at least one candidate available cache node from the multiple available cache nodes based on the training data; and a control module, configured to determine a target available cache node from the at least one candidate available cache node based on the cache priority, and control the target available cache node to preload training data.

[0007] A third aspect of this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described training data preloading method.

[0008] A fourth aspect of this disclosure provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the above-described training data preloading method.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided that, when executed by a processor, implements the steps of the above-described method for preloading training data.

[0010] The above-mentioned at least one technical solution adopted in the embodiments of this disclosure can achieve the following beneficial effects: when a failure of the current cache node is detected, the node information of multiple available cache nodes in the target cluster and the training data of the model training task are obtained; the cache priority of loading training data of multiple available cache nodes is determined according to the node information of multiple available cache nodes respectively; at least one candidate available cache node is determined from multiple available cache nodes according to the training data; the target available cache node is determined from at least one candidate available cache node according to the cache priority, and the target available cache node is controlled to preload training data.

[0011] As can be seen, the embodiments of this disclosure can trigger a multi-stage processing flow immediately upon detecting a failure of the current cache node. First, the node information of available cache nodes in the target cluster can be obtained; then, the cache priority of each available cache node is determined through the node information; next, candidate available cache nodes with caching potential are selected based on training data characteristics; finally, the optimal target available cache node is selected based on cache priority to perform the preloading operation of training data. In this way, intelligent node selection can replace manual intervention, avoiding cache resource overload; at the same time, the priority-driven preloading strategy optimizes the data access path of the faulty current cache node from remote reading to local cache reading, reducing the data reading latency and improving efficiency; furthermore, the preloading design completes the deployment of training data in advance, significantly shortening the time for training tasks to wait for data loading and reducing the idle period of the training process. Especially in scenarios with high requirements for data timeliness, such as large-scale training, it can significantly improve the fault tolerance, resource utilization, and business stability of the target cluster. Attached Figure Description

[0012] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 A schematic diagram of the structure of a cache coordination system provided as an exemplary embodiment of this disclosure;

[0014] Figure 2 A flowchart illustrating a method for preloading training data provided as an exemplary embodiment of this disclosure;

[0015] Figure 3 This is a schematic diagram of the structure of a training data preloading device provided in an embodiment of the present disclosure;

[0016] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0017] Figure 5 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure;

[0018] Figure 6 A schematic diagram of a computer program product provided according to an embodiment of this disclosure. Detailed Implementation

[0019] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0020] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0021] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0022] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0023] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0024] With the rapid development of artificial intelligence technology, the demand for model training in cloud environments continues to grow, especially in large-scale model training scenarios, which are characterized by high training costs, large computing node scale, and long training cycles. Typically, during model training, training data is frequently accessed, during which GPU nodes are usually idle, leading to reduced computing resource utilization and significant resource waste. Here, a GPU node, short for Graphics Processing Unit Node, refers to a hardware unit that integrates graphics processing unit hardware and possesses independent computing, storage, and network communication capabilities. It is a core component for building GPU node clusters or single-machine high-performance computing systems.

[0025] Among related technologies, low-cost cloud storage services, such as object storage, can be provided. Using object storage as the underlying storage architecture during model training helps to further control costs. However, object storage is usually located in a different physical or logical network area from the model training computing cluster. This means that when GPU nodes directly read data from object storage, they need to access it remotely over the network. This not only introduces high network transmission overhead but also results in significant data read latency, becoming one of the bottlenecks in training performance.

[0026] To address the aforementioned issues, related technologies employ distributed caching, which caches a portion of the data from remote storage onto local cache nodes. This allows GPU nodes to directly read training data from the local cache, effectively converting remote reads into local reads and writes and significantly improving I / O performance. However, with distributed caching, if the required training data is not yet cached, it still needs to be read from remote storage before being stored in the cache. In this case, the speed of reading training data can be even slower than directly accessing remote storage. To address the issue of low performance on the first read, manual pre-configuration can be used to pre-load the required dataset into the distributed caching system before the training task is executed. This ensures that data can be retrieved from the local cache on the first read, avoiding remote access latency.

[0027] However, manual pre-configuration still has shortcomings, especially in multi-task parallel and fault-tolerant scenarios, where its flexibility and automation are limited. First, in model training scenarios, multiple model training tasks often run concurrently within the same cluster, with each task occupying only a portion of the cluster's cache nodes. If all the training data required for all model training tasks is manually preloaded into the distributed cache system, the limited cache capacity may not be able to accommodate all the training data, leading to cache resource contention and decreased efficiency. Second, model training is typically time-consuming, during which cache nodes may fail. To ensure the continuous operation of model training tasks, they are usually rescheduled to other cache nodes to resume training after a cache node fails. However, cache node failures are unpredictable, and newly assigned cache nodes often fail to pre-load the required training data. Therefore, when the cache node first reads the training data, it still faces high access latency and slow read speeds, affecting the recovery efficiency of the training task.

[0028] To address the aforementioned issues, this disclosure provides a method for preloading training data, comprising: when a failure of the current cache node is detected, acquiring node information of multiple available cache nodes in the target cluster and training data for a model training task; determining the cache priority of loading training data for the multiple available cache nodes based on the node information of each available cache node; identifying at least one candidate available cache node from the multiple available cache nodes based on the training data; identifying a target available cache node from the at least one candidate available cache node based on the cache priority, and controlling the target available cache node to preload training data.

[0029] As can be seen, the embodiments of this disclosure can trigger a multi-stage processing flow immediately upon detecting a failure of the current cache node. First, the node information of available cache nodes in the target cluster can be obtained; then, the cache priority of each available cache node is determined through the node information; next, candidate available cache nodes with caching potential are selected based on training data characteristics; finally, the optimal target available cache node is selected based on cache priority to perform the preloading operation of training data. In this way, intelligent node selection can replace manual intervention, avoiding cache resource overload; at the same time, the priority-driven preloading strategy optimizes the data access path of the faulty current cache node from remote reading to local cache reading, reducing the data reading latency and improving efficiency; furthermore, the preloading design completes the deployment of training data in advance, significantly shortening the time for training tasks to wait for data loading and reducing the idle period of the training process. Especially in scenarios with high requirements for data timeliness, such as large-scale training, it can significantly improve the fault tolerance, resource utilization, and business stability of the target cluster.

[0030] The training data preloading method provided in this disclosure can be executed by a terminal or by a chip applied to the terminal.

[0031] For example, the aforementioned terminals may include one or more of the following: mobile phones, tablets, wearable devices, in-vehicle devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, handheld computers (PDAs), and wearable devices based on augmented reality (AR) and / or virtual reality (VR) technologies. They may also include, but are not limited to, remote control devices, wearable devices, streetlights, home appliances, and other smart terminals. This disclosure does not impose specific limitations on these aspects.

[0032] Figure 1 This is a schematic diagram of the structure of a cache coordination system provided as an exemplary embodiment of the present disclosure. Figure 1 As shown, it includes:

[0033] The intelligent computing platform 101, as the upper-level overall planning module, is used to manage the scheduling and resource coordination of model training tasks, and is the brain of the entire process; the cache coordinator 102 is used to undertake the functions of cache strategy formulation, cache node status management, and coordination of cache resource allocation; the training node 103 includes the cache node 1031, which is a temporary warehouse for training data and is used to store data; the object storage 104 provides data support for the training node.

[0034] like Figure 1As shown, when the intelligent computing platform 101 distributes model training tasks to each training node 103, the cache coordinator 102 can communicate bidirectionally with the intelligent computing platform 101 and the training node 103 to dynamically allocate cache resources. The training node 103 reads and writes training data from the object storage 104 as needed, and the cache node 1031 uses caching strategies to accelerate the access to training data, ensuring the efficient execution of model training tasks, realizing a closed-loop process of task scheduling, cache coordination and data access, and adapting to the needs of large-scale distributed model training for high-concurrency and low-latency data access.

[0035] Figure 2 This is a flowchart illustrating a method for preloading training data, provided as an exemplary embodiment of this disclosure. Figure 2 As shown, the embodiments of this disclosure can be applied to Figure 1 The cache coordinator shown includes the following specific methods for preloading training data:

[0036] S201, if a failure is detected in the current cache node, obtain the node information of multiple available cache nodes in the target cluster and the training data included in the current cache node. It should be understood that the training data here refers to the training data required for the model training task, and one training data set may include one or more training sub-data sets corresponding to one model training task.

[0037] In some embodiments, node information may include at least one of the following: remaining cache space of available cache nodes, cached data, the amount of cached data, the number of failures, and the time of the last anomaly; while the training data may be the data corresponding to the model training task being executed by the currently faulty cache node. Based on this, after the cache coordinator receives a fault signal from the current cache node, it can immediately obtain the node information of all available cache nodes in the target cluster, and then use one of the multiple available cache nodes to pull the training data. That is, it can use available cache nodes to directly take over the data needs of the faulty cache node, minimizing the training interruption time and improving the efficiency of model training.

[0038] S202, based on the node information of multiple available cache nodes, determines the cache priority for loading training data on multiple available cache nodes.

[0039] In some embodiments, the cache priority for loading training data on any available cache node can be determined based on the remaining cache space of the available cache node, the cached data, the amount of cached data, the number of failures, and the time of the last exception.

[0040] Specifically, the cache priority when the i-th available cache node loads the j-th training data can be defined as... Here, The calculation formula is as follows:

[0041]

[0042] in, for and The result of the sum and the result after the sigmoid function transformation can be guaranteed by the sigmoid function transformation. The value is within the range of (0, 1), which makes it convenient to set the priority threshold. The node information includes one of the following: remaining cache space of the available cache node, cached data, the amount of cached data, the number of failures, and the time of the last anomaly, where k = 1, 2, 3, 4. Here, the Sigmoid function is a common sigmoid activation function widely used in machine learning (especially logistic regression and neural networks). Its core function is to map input values ​​from any real number field to the interval [0, 1], achieving "numerical normalization" or "probabilistic output".

[0043] And for These can be defined separately, and the specific calculations are as follows:

[0044]

[0045] in, This indicates the impact of the remaining cache space of available cache nodes on cache priority. Here, the more remaining cache space, the greater the likelihood that an available cache node will be used. This is a scaling factor, which can be used as a custom value. The proportion of cache priority; This represents the remaining available cache space size for the available cache nodes. This represents the total cache space size available for cache nodes.

[0046]

[0047] in, This indicates the impact of the amount of cached data on available cache nodes on cache priority; This is a scaling factor, which can be used as a custom value. The proportion of cache priority; This indicates the number of model training tasks that have been cached on the available cache node. Here, if the available cache node has already cached a large number of training tasks, other available cache nodes can be searched instead of loading all training datasets into the same available cache node. To score the importance of the training data, if the importance score of a certain cached data in an available cache node is very high, the influence of other cached data can be ignored, and the cached data with the higher importance score can still be loaded on that available cache node.

[0048]

[0049] in, This indicates the impact of the last time an available cache node experienced an exception on its priority. This is a scaling factor, which can be used as a custom value. The proportion of cache priority; This is the difference between the time of the last failure of the available cache node and the current time.

[0050]

[0051] in, This indicates the impact of the number of failures of available cache nodes on priority; This is a scaling factor, which can be used as a custom value. In terms of cache priority, The cumulative number of failures for this available cache node.

[0052] Based on this, the priority calculation mechanism of this disclosure integrates four core indicators: the remaining cache space of available cache nodes, the amount of cached data, the time of the last failure, and the number of failures. After normalization by the sigmoid function, a priority value in the range of (0,1) is formed. Prioritize selecting available cache nodes with sufficient remaining cache space to avoid cache overflow, and also through... To prevent training data from being overly concentrated on a single available cache node, and also through and We prefer to select stable and available cache nodes with long failure intervals and few failures, while allowing the use of a scaling factor. By flexibly adjusting the weights of various factors to adapt to different scenario requirements, the scientific quantification of the priority of loading training data on available cache nodes was finally achieved. This provides an accurate decision-making basis for the efficient migration of training data after a failure, ensuring both the reliability and efficiency of data loading and the balanced utilization of cluster resources.

[0053] S203, determine at least one candidate available cache node from multiple available cache nodes based on the training data.

[0054] In some embodiments, the priority of obtaining all available cache nodes can be determined. Then, priority thresholds can be set to filter out... Available cache nodes with a priority greater than the threshold are selected as candidate available cache nodes. If the selection result is empty, the top 3 nodes with the highest priority are selected as candidate available cache nodes. Alternatively, the cache priority of available cache nodes that have already run training data can be set to 0, and then available cache nodes that do not contain training data can be selected as candidate available cache nodes from the remaining available cache nodes. Based on this, highly suitable candidate available cache nodes can be quickly identified based on a quantified cache priority index, avoiding the waste of resources by low-priority candidate available cache nodes.

[0055] S204, determine the target available cache node from at least one candidate available cache node according to the cache priority, and control the target available cache node to preload training data.

[0056] In some embodiments, when there is only one candidate available cache node, the candidate available cache node can be determined as the target available cache node; when there are multiple candidate available cache nodes, the candidate available cache node with the highest cache priority can be determined as the target available cache node based on the cache priority of at least one candidate available cache node.

[0057] Based on this, the embodiments of this disclosure can trigger a multi-stage processing flow immediately upon detecting a failure of the current cache node. First, the node information of available cache nodes in the target cluster can be obtained; then, the cache priority of each available cache node is determined through the node information; next, candidate available cache nodes with caching potential are selected based on training data characteristics; finally, the optimal target available cache node is selected according to the cache priority to perform the preloading operation of training data. In this way, intelligent node selection can replace manual intervention, avoiding cache resource overload; at the same time, the priority-driven preloading strategy optimizes the data access path of the faulty current cache node from remote reading to local cache reading, reducing the data reading latency and improving efficiency; furthermore, the preloading design completes the deployment of training data in advance, significantly shortening the time for training tasks to wait for data loading and reducing the idle period of the training process. Especially in scenarios with high requirements for data timeliness, such as large-scale training, it can significantly improve the fault tolerance, resource utilization, and business stability of the target cluster.

[0058] In some embodiments, the training data includes multiple training data, and the preloading method for the training data further includes: determining the loading priority of the multiple training data according to the importance scores of the multiple training data respectively; and sequentially controlling the target available cache nodes corresponding to the multiple training data to preload the multiple training data in order according to the loading priority of the multiple training data.

[0059] Specifically, multiple training data points correspond to multiple model training tasks, and the importance scores of the training data can be assigned by relevant personnel based on the importance of the model training tasks. For example, suppose the multiple training data points include the first training data. Second training data and third training data At this point, the importance scores can be ranked as follows: Based on the importance ranking, the second training data has the highest importance score. Therefore, it can be processed first. This involves first determining the caching priority of multiple available caching nodes for loading the second training data. Then, based on the second training data, at least one candidate available caching node is selected from the multiple available caching nodes. This yields a matrix of candidate available caching nodes corresponding to the second training data. Finally, based on the caching priorities of the multiple candidate available caching nodes, the candidate available caching node with the highest caching priority is determined as the target available caching node. The specific matrix of candidate available caching nodes is as follows:

[0060]

[0061] in, This indicates the caching priority of the first candidate available cache node for the second training data; This indicates the caching priority of the second candidate available cache node for the second training data; This indicates the caching priority of the i-th candidate available cache node for the second training data.

[0062] When there are multiple training data sets, the above method can be used to determine the target available cache nodes for each different training data set. Then, the training data sets are loaded sequentially according to their loading priority. This approach, by first determining the loading priority based on the importance score of the training data, and then controlling the preloading of the corresponding target available cache nodes in this order, ensures that core training data receives cache resources first, reducing the risk of training blockage due to loading delays. Ordered loading also avoids network bandwidth contention and node resource conflicts caused by parallel transmission of multiple data sets, improving overall data loading efficiency. Furthermore, dynamically adjusting the loading order based on the importance of the training data maximizes the availability of high-value data with limited cache resources, making the fault recovery process more aligned with the priority ranking of actual training needs, further optimizing the rationality of cache resource allocation and the continuity of training tasks.

[0063] In some embodiments, node information includes cached data in available cache nodes, and determining at least one candidate available cache node from multiple available cache nodes based on training data includes: determining an available cache node as a candidate available cache node when the cached data in an available cache node does not match the training data.

[0064] Specifically, by selecting available cache nodes whose cached data does not match the training data as candidate available cache nodes, it is possible to effectively avoid repeatedly loading training data into available cache nodes that already have the same data. This reduces the storage space occupied by redundant caches, lowers the network transmission overhead of data replication, and improves the utilization efficiency of cache resources by avoiding duplicate storage. At the same time, this selection logic ensures the distributed storage of training data in the cluster, reduces the risk of losing critical data due to the failure of a single node, and makes the data distribution of the caching system more reasonable and reliable, providing better infrastructure support for data access in subsequent training tasks.

[0065] In some embodiments, the node information includes cached data and remaining cache space in the available cache nodes, and the preloading method for training data further includes: for any candidate available cache node, when the remaining cache space of the candidate available cache node is less than the amount of training data, determining the cached data to be removed from the cached data in the candidate available cache node according to the amount of training data; removing the cached data to be removed, and re-determining the cache priority when the candidate available cache node loads the training data.

[0066] Specifically, assume the remaining cache space of a candidate available cache node is 50GB, while the amount of training data to be loaded is 80GB, a difference of 30GB. In this case, we can analyze the 100GB of data already cached by the candidate available cache node. For example, this candidate available cache node includes 40GB of cached data for model A training task, 30GB of cached data for model B training task, and 30GB of cached data for model C training task. Based on rules such as data access frequency and the execution priority of model training tasks, we can determine to remove the cached data for model B training task, which has the lowest access frequency. After removal, the remaining cache space of the node becomes 80GB, meeting the training data loading requirements. Then, the cache priority of the candidate available cache node is recalculated. Due to the change in the amount of cached data, the corresponding formula above... The value also needs to be adjusted accordingly.

[0067] As can be seen, the embodiments of this disclosure can solve the problem of insufficient remaining capacity of candidate available cache nodes by dynamically cleaning up low-priority cache data to free up space, ensuring that training data can be loaded smoothly and improving the elastic utilization capability of cache resources. At the same time, the removal operation is precisely executed in combination with data characteristics, reducing the impact on high-value cached data. Recalculating priorities ensures the accuracy of decision-making after node status changes. This not only avoids the failure of candidate available cache nodes due to space limitations, but also balances the caching needs of new and old data through fine-grained space management, enhancing the system's adaptability to dynamic storage pressure.

[0068] In some embodiments, the cached data includes data corresponding to multiple cache tasks. Determining the cached data to be removed from the cached data in the candidate available cache nodes based on the amount of training data includes: determining the replacement cost when the data corresponding to each cache task is removed based on the importance score and data volume of each cache task; determining the removal priority of each cache task based on the replacement cost of each cache task; and determining at least one cache task to be removed from the multiple cache tasks based on the data volume and removal priority of each cache task, wherein the data corresponding to the at least one cache task to be removed is the cached data to be removed. Here, the replacement cost may include at least one of the time cost, economic cost, and time cost, economic cost, etc., of removing the data of the cache task to be removed.

[0069] Specifically, the replacement cost of cache task t on candidate available cache node y can be calculated using the following formula. The specific calculation formula is as follows:

[0070]

[0071] in, The size of the dataset corresponding to caching task t that has already been cached on the candidate available cache node y; It is the remaining cache space size that the candidate available cache node y can cache data for; is the importance score of the cached task t already cached by the candidate available cache node y; r is the cache replacement cost coefficient; R is a value that varies depending on the underlying storage system. Here, the candidate available cache node y belongs to the available cache node i.

[0072] Based on this, a multi-dimensional decision-making chain encompassing importance scoring, data volume, replacement cost, and removal priority enables refined management of cached data removal. Its core value lies in breaking through the traditional cache eviction logic based solely on data volume or access frequency. By quantifying the importance of cached tasks into replacement costs, it avoids the resource waste that can result from blindly deleting large volumes of low-value data, while also preventing inefficient cache space usage caused by retaining low-priority, small-volume data. This hierarchical decision-making mechanism prioritizes the retention of highly important data within limited cache node resources. Simultaneously, through a comprehensive evaluation of data volume and priority, it ensures that removal operations meet the needs of new data storage while minimizing overall cache replacement costs, ultimately improving the resource utilization and data service efficiency of the caching system.

[0073] In some embodiments, the replacement cost of removing at least one cached data to be removed is determined, and the corresponding use value after loading the training data is determined; if the replacement cost of at least one cached data to be removed is less than the use value, the cached data to be removed is removed; or if the replacement cost of at least one cached data to be removed is greater than or equal to the use value, a target available cache node is re-determined from at least one candidate available cache node according to cache priority. Here, the use value may include the economic value generated after the training data is loaded and used.

[0074] Specifically, suppose we need to load the second training data. At this point, for a candidate available cache node y, if the cache task t corresponds to... If <M2, then the cache priority can be determined based on the cached data corresponding to cache task t in the candidate available cache node y. The correction involves removing the cached data corresponding to task 2 from the candidate available cache node y, and then using the freed-up cache space to load the second training data, where M2 represents the value of the second training data.

[0075] Based on this, it can be determined that... After filtering out all cache tasks to be removed using the M2 condition, the candidate available cache node y can be recalculated when loading the second training data. Specifically, after the remaining cache space of candidate available cache node y changes, it is necessary to modify it synchronously. and The value is as follows:

[0076]

[0077] in, This is the remaining cache space size of candidate available cache nodes after releasing all cache tasks to be removed. This is the number of all cache tasks to be removed. Score the importance of the second training data.

[0078] In some embodiments, after removing the cached data to be removed, the cache priority of the corresponding candidate available cache node can be re-determined. Further, the cache priority of each candidate available cache node whose remaining cache space is less than the amount of training data can be recalculated according to the above content. Then, the cache priorities of all candidate available cache nodes are reordered according to the re-determined cache priorities, so as to re-determine the target cache node with the highest cache priority as the target cache node of the second training data, and start the process of loading the second training data.

[0079] In some embodiments, the above method may be used only for the target cache node. That is, after the target available cache node is determined from at least one of the candidate available cache nodes according to the cache priority, the above method is used to determine the cache data to be removed in the target available cache node when the remaining cache space of the target available cache node is less than the amount of training data. Then the cache data to be removed is removed, and the training data is loaded using the target available cache node.

[0080] In some embodiments, the replacement cost when data corresponding to each cache task is removed is determined based on the importance score and data volume corresponding to each cache task, and the data loading process is initiated. Subsequent cache tasks need to consider previously started loading tasks when making secondary adjustments to cache priorities. For example, for the second training data corresponding to cache task 2, available cache node 5 can be selected to load the training data. Therefore, during the adjustment... When and It needs to be changed to:

[0081]

[0082] S2 is the second training data. The corresponding amount of data.

[0083] In some embodiments, as time goes on, new model training tasks are continuously undertaken, and the node information of the cache nodes will change. Therefore, the cache coordinator can be controlled to periodically execute the process of the preloading method for training data in the embodiments of this disclosure to achieve data preloading.

[0084] Based on this, the embodiments of this disclosure construct a dynamic and elastic cache management decision-making logic by introducing a quantitative comparison mechanism between replacement cost and usage value, significantly improving the rationality of cache resource scheduling. When the cost of removing data to be deleted is lower than the usage value of new data, the removal operation is performed to achieve optimized resource allocation; while when the replacement cost is too high or not worthwhile, a priority re-evaluation mechanism for candidate available cache nodes is initiated to avoid irrational cache replacement. This two-layer decision-making mode not only prevents the sacrifice of high-cost cache resources for low-value data, but also ensures the flexibility of cache space adjustment through secondary priority screening, ultimately maximizing the overall utility of the cache system under resource constraints and balancing short-term storage needs with long-term resource utilization efficiency.

[0085] As can be seen, the embodiments of this disclosure can trigger a multi-stage processing flow immediately upon detecting a failure of the current cache node. First, the node information of available cache nodes in the target cluster can be obtained; then, the cache priority of each available cache node is determined through the node information; next, candidate available cache nodes with caching potential are selected based on training data characteristics; finally, the optimal target available cache node is selected based on cache priority to perform the preloading operation of training data. In this way, intelligent node selection can replace manual intervention, avoiding cache resource overload; at the same time, the priority-driven preloading strategy optimizes the data access path of the faulty current cache node from remote reading to local cache reading, reducing the data reading latency and improving efficiency; furthermore, the preloading design completes the deployment of training data in advance, significantly shortening the time for training tasks to wait for data loading and reducing the idle period of the training process. Especially in scenarios with high requirements for data timeliness, such as large-scale training, it can significantly improve the fault tolerance, resource utilization, and business stability of the target cluster.

[0086] The foregoing primarily describes the solutions provided by the embodiments of this disclosure from the perspective of the server. It is understood that, in order to implement the above functions, the server includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0087] This disclosure embodiment can divide the server into functional units according to the above method example. For example, it can divide each function into separate functional modules, or it can integrate two or more functions into one management module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this disclosure embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0088] In the case of dividing each functional module according to its corresponding functions, an exemplary embodiment of this disclosure provides a training data preloading device, which can be a server or a chip applied to a server. Figure 3 This is a schematic diagram of a preloading device for training data provided in one embodiment of this disclosure. Figure 3 As shown, the training data preloading device 300 includes:

[0089] The acquisition module 301 is used to acquire node information of multiple available cache nodes in the target cluster and training data included in the current cache node when a failure of the current cache node is detected.

[0090] The first determining module 302 is used to determine the cache priority of loading the training data by the multiple available cache nodes based on the node information of the multiple available cache nodes.

[0091] The second determining module 303 is used to determine at least one candidate available cache node from the plurality of available cache nodes based on the training data.

[0092] The control module 304 is used to determine a target available cache node from the at least one candidate available cache node according to the cache priority, and control the target available cache node to preload the training data.

[0093] In one alternative approach, the training data includes multiple datasets, and the method further includes: determining the loading priority of the multiple training data based on the importance scores of the multiple training data respectively; sequentially controlling the target available cache nodes corresponding to the multiple training data, and preloading the multiple training data according to the loading priority order of the multiple training data.

[0094] In one alternative approach, the node information includes cached data in the available cache nodes, and determining at least one candidate available cache node from the plurality of available cache nodes based on the training data includes: determining the available cache node as a candidate available cache node if the cached data in the available cache node does not match the training data.

[0095] In an optional manner, the node information includes cached data and remaining cache space in the available cache nodes. The method further includes: for any candidate available cache node, when the remaining cache space of the candidate available cache node is less than the amount of training data, determining cached data to be removed from the cached data in the candidate available cache node based on the amount of training data; removing the cached data to be removed, and re-determining the cache priority when the candidate available cache node loads the training data.

[0096] In one optional approach, the cached data includes data corresponding to multiple cache tasks. The step of determining the cached data to be removed from the cached data in the candidate available cache nodes based on the amount of training data includes: determining the replacement cost when the data corresponding to each cache task is removed based on the importance score and data amount of each cache task; determining the removal priority of each cache task based on the replacement cost of each cache task; and determining at least one cache task to be removed from the multiple cache tasks based on the data amount and removal priority of each cache task, wherein the data corresponding to the at least one cache task to be removed is the cached data to be removed.

[0097] In one alternative approach, the replacement cost of removing the at least one cached data to be removed is determined, and the corresponding usage value after loading the training data is determined; if the replacement cost of the at least one cached data to be removed is less than the usage value, the cached data to be removed is removed; or if the replacement cost of the at least one cached data to be removed is greater than or equal to the usage value, a target available cache node is re-determined from at least one candidate available cache node according to cache priority.

[0098] This disclosure also provides an electronic device, including: at least one processor; a memory for storing at least one processor-executable instruction; wherein the at least one processor is used to execute the instruction to implement the steps of the method disclosed in this disclosure.

[0099] Figure 4 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. Figure 4 As shown, the electronic device 400 includes at least one processor 401 and a memory 402 coupled to the processor 401, which can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0100] The processor 401 described above can also be called a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the hardware of the processor 401 or by instructions in software form. The processor 401 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 402, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 401 reads information from the memory 402 and, in conjunction with its hardware, completes the steps of the method described above.

[0101] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example, Figure 5 The computer system 500 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those mentioned above. Figure 5 This is a schematic diagram of the structure of a computer system provided in an embodiment of the present disclosure.

[0102] Computer system 500 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of this disclosure described and / or claimed herein.

[0103] like Figure 5As shown, the computer system 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the computer system 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0104] Multiple components in the computer system 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 can be any type of device capable of inputting information into the computer system 500. The input unit 506 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 508 may include, but is not limited to, a hard disk and an optical disk. The communication unit 509 allows the computer system 500 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0105] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU node), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 502 and / or communication unit 509. In some embodiments, the computing unit 501 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0106] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0107] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0109] Figure 6 This is a schematic diagram of a computer program product provided according to an embodiment of the present disclosure. Figure 6 As shown, the computer program product 600 includes a computer program 601, which, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0110] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0112] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0113] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0114] The above description is merely an illustration of some embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0115] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for preloading training data, characterized in that, include: If a failure is detected in the current cache node, obtain the node information of multiple available cache nodes in the target cluster and the training data included in the current cache node; wherein, the node information includes at least one of the following: remaining cache space, amount of cached data, number of failures, and time of the last anomaly; Based on the node information of the multiple available cache nodes, determine the cache priority of the multiple available cache nodes for loading the training data; Based on the training data, at least one candidate available cache node is determined from among the multiple available cache nodes; Based on the cache priority, a target available cache node is determined from the at least one candidate available cache node, and the target available cache node is controlled to preload the training data; The node information includes the cached data and remaining cache space in the available cache nodes, and the method includes: For any candidate available cache node, if the remaining cache space of the candidate available cache node is less than the amount of training data, the cache data to be removed is determined from the cached data in the candidate available cache node according to the amount of training data. Remove the cached data to be removed, and redetermine the cache priority when the candidate available cache nodes load the training data; The target available cache node is determined based on the cache priority when loading the training data from the re-determined candidate available cache nodes, and the target available cache node is controlled to preload the training data. The formula for calculating the cache priority of the available cache nodes loading the training data is as follows: in, This indicates the cache priority when the i-th available cache node loads the j-th training data. The node information is represented by k=1,2,3,4; i represents the identifier of the available cache node, and j represents the identifier of the training data; When k=1 The calculation formula is: in, This indicates the impact of the remaining cache space of the available cache nodes on cache priority; This is a scaling factor, used for custom applications. The proportion of cache priority; The remaining available cache space size of the available cache nodes; The total cache space size for the available cache nodes; When k=2 The calculation formula is: in, This indicates the impact of the amount of cached data in the available cache nodes on cache priority; This is a scaling factor, used for custom applications. The proportion of cache priority; This indicates the number of model training tasks already cached on the available cache nodes; Score the importance of the training data; When k=3 The calculation formula is: in, This indicates the impact of the last time an exception occurred on the priority of the available cache node; This is a scaling factor, used for custom applications. The proportion of cache priority; The difference between the time of the last failure of the available cache node and the current time; When k=4 The calculation formula is: in, This indicates the impact of the number of failures of the available cache nodes on the priority; This is a scaling factor, used for custom applications. In terms of cache priority, The cumulative number of failures for the available cache nodes.

2. The method according to claim 1, characterized in that, The training data includes multiple datasets, and the method further includes: The loading priority of the training data is determined based on the importance scores of the training data. The target available cache nodes corresponding to multiple training data are controlled sequentially, and the multiple training data are preloaded according to the loading priority order of the multiple training data.

3. The method according to claim 1, characterized in that, The node information includes cached data in the available cache nodes, and the step of determining at least one candidate available cache node from the plurality of available cache nodes based on the training data includes: If the cached data in the available cache node does not match the training data, the available cache node is determined as a candidate available cache node.

4. The method according to claim 1, characterized in that, The cached data includes data corresponding to multiple cache tasks. The step of determining the cached data to be removed from the cached data in the candidate available cache nodes based on the amount of training data includes: Based on the importance score and data volume corresponding to each cache task, determine the replacement cost when the data corresponding to each cache task is removed; The removal priority of each cache task is determined based on the replacement cost of each cache task; Based on the data volume and removal priority of each cache task, at least one cache task to be removed is determined from the plurality of cache tasks, wherein the data corresponding to the at least one cache task to be removed is the cache data to be removed.

5. The method according to claim 4, characterized in that, The method further includes: Determine the replacement cost of removing the at least one cached data to be removed, and determine the corresponding use value after loading the training data; If the replacement cost of at least one cached data item to be removed is less than the value of its use, then the cached data item to be removed is removed; or If the replacement cost of the at least one cached data to be removed is greater than or equal to the value of use, the target available cache node is re-determined from at least one candidate available cache node according to the cache priority.

6. A preloading device for training data, characterized in that, include: The acquisition module is used to acquire node information of multiple available cache nodes in the target cluster and training data included in the current cache node when a failure of the current cache node is detected; wherein, the node information includes at least one of the following: remaining cache space, amount of cached data, number of failures, and time of the last anomaly; The first determining module is used to determine the cache priority of loading the training data by the multiple available cache nodes based on the node information of the multiple available cache nodes respectively; The second determining module is used to determine at least one candidate available cache node from the plurality of available cache nodes based on the training data; A control module is configured to determine a target available cache node from the at least one candidate available cache nodes according to the cache priority, and control the target available cache node to preload the training data; The control module is further configured to, for any candidate available cache node, when the remaining cache space of the candidate available cache node is less than the amount of training data, determine the cache data to be removed from the cached data in the candidate available cache node according to the amount of training data; remove the cache data to be removed, and re-determine the cache priority when the candidate available cache node loads the training data; determine the target available cache node according to the re-determined cache priority when the candidate available cache node loads the training data, and control the target available cache node to preload the training data; The formula for calculating the cache priority of the available cache nodes loading the training data is as follows: in, This indicates the cache priority when the i-th available cache node loads the j-th training data. The node information is represented by k=1,2,3,4; i represents the identifier of the available cache node, and j represents the identifier of the training data; When k=1 The calculation formula is: in, This indicates the impact of the remaining cache space of the available cache nodes on cache priority; This is a scaling factor, used for custom applications. The proportion of cache priority; The remaining available cache space size of the available cache nodes; The total cache space size for the available cache nodes; When k=2 The calculation formula is: in, This indicates the impact of the amount of cached data in the available cache nodes on cache priority; This is a scaling factor, used for custom applications. The proportion of cache priority; This indicates the number of model training tasks already cached on the available cache nodes; Score the importance of the training data; When k=3 The calculation formula is: in, This indicates the impact of the last time an exception occurred on the priority of the available cache node; This is a scaling factor, used for custom applications. The proportion of cache priority; The difference between the time of the last failure of the available cache node and the current time; When k=4 The calculation formula is: in, This indicates the impact of the number of failures of the available cache nodes on the priority; This is a scaling factor, used for custom applications. In terms of cache priority, The cumulative number of failures for the available cache nodes.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.

Images