Checkpoint deployment method, system, product and device for heterogeneous computing system

Abstract

The invention discloses a check point deployment method, system, product and device for a heterogeneous computing system, and relates to the technical field of computers. According to the demand parameter of each heterogeneous computing power device, calculating a fault recovery time-consuming expected value corresponding to each storage device when the check point is deployed on each storage device, so as to balance the influence of the recovery of the check point corresponding to each demand parameter of each heterogeneous computing power device. And determining a preliminary first check point deployment relationship according to the demand parameter of each heterogeneous computing power device and the fault recovery time-consuming expected value, so as to realize the predictability of local optimal selection from the corresponding deployment check point to each storage device for the characteristic parameter of each heterogeneous computing power device in the deployment process. According to the preliminary first check point deployment relation, the second check point deployment relation is determined through optimization of the deployment positions of the heterogeneous computing power devices, the possibility of finding a global optimal solution by effectively jumping out of a local optimal solution is increased, and the execution efficiency of a training task is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system, product and device for deploying checkpoints in a heterogeneous computing system. Background Art

[0002] In a heterogeneous computing system, the checkpoints corresponding to the model states that are saved periodically during the model training process are saved in the extended memory of the heterogeneous computing system, such as an external solid state drive (SSD) or hard disk drive (HDD) storage. When a failure occurs in the heterogeneous computing device corresponding to the model training task, the checkpoint is placed in a fixed position. During the checkpoint recovery process, if the input and output speed of the current external memory is slow, and the communication rate between the extended memory and the heterogeneous computing device is also slow, the checkpoint recovery rate will be reduced, thereby affecting the execution efficiency of the model training task.

[0003] Therefore, how to increase the checkpoint recovery rate to improve the execution efficiency of training tasks is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] The purpose of the present invention is to provide a checkpoint deployment method, system, product and device for a heterogeneous computing system to solve the problem that when a failure occurs in a heterogeneous computing device corresponding to a model training task, the corresponding checkpoint recovery efficiency is low, resulting in reduced execution efficiency of the model training task.

[0005] In order to solve the above technical problems, the present invention provides a checkpoint deployment method for a heterogeneous computing system, comprising: Obtain the required parameters of heterogeneous computing devices and storage devices of heterogeneous computing systems; Deploy a checkpoint for each storage device according to the demand parameters of each heterogeneous computing device to determine the expected time for failure recovery of each storage device; Determine the first checkpoint deployment relationship according to the demand parameters of each heterogeneous computing device and the expected time consumption of the fault recovery; The second checkpoint deployment relationship is determined by optimizing the deployment positions of the heterogeneous computing devices within the first checkpoint deployment relationship, so as to deploy the checkpoints of the target heterogeneous computing devices of the training task according to the second checkpoint deployment relationship.

[0006] On the one hand, a checkpoint is deployed for each storage device according to the demand parameters of each heterogeneous computing device to determine the expected time consumption for failure recovery of each storage device, including: Obtaining a failure probability parameter of each heterogeneous computing device and a resource parameter of each storage device corresponding to each heterogeneous computing device; The expected time consumption for failure recovery corresponding to each storage device when each storage device corresponding to each heterogeneous computing device deploys the checkpoint is determined according to the failure probability parameter of each heterogeneous computing device and the resource parameter.

[0007] On the other hand, the resource parameters at least include a latency parameter, a bandwidth parameter, and a current memory occupied space of each heterogeneous computing device corresponding to each storage device when each heterogeneous computing device executes a checkpoint corresponding to a training task; determining the expected time consumption for failure recovery corresponding to each storage device when each heterogeneous computing device deploys the checkpoint corresponding to each storage device according to the failure probability parameter of each heterogeneous computing device and the resource parameters, including: Determine the transmission time according to the current memory occupied space and the bandwidth parameter; Determine the total delay parameter corresponding to each of the heterogeneous computing devices to each storage device according to the transmission time and delay parameter; The expected time consumption for failure recovery corresponding to each storage device when each storage device corresponding to each heterogeneous computing device deploys the checkpoint is determined according to the total delay parameter and the failure probability parameter.

[0008] On the other hand, the process of determining the failure probability parameter of each of the heterogeneous computing devices includes: Obtain the empirical failure probability parameters of the current heterogeneous computing power equipment; Determine the historical failure probability parameters of the current heterogeneous computing power equipment within a preset time interval; Determine a corresponding first weight parameter according to the amount of maintenance data of the historical fault probability parameter; Determine a second weight parameter of the empirical fault probability parameter according to the first weight parameter; The failure probability parameter of the current heterogeneous computing power device is determined according to the historical failure probability parameter, the first weight parameter, the empirical failure probability parameter and the second weight parameter.

[0009] On the other hand, the process of determining the latency parameter of each storage device corresponding to each of the heterogeneous computing devices includes: Obtain the read / write latency and access latency from the current heterogeneous computing device to each storage device; Determine a first delay parameter according to the read / write delay and the access delay; Obtain the query rate of the current heterogeneous computing device under each concurrent condition to determine the corresponding second delay parameter; Taking an average value of each of the second delay parameters to obtain a third delay parameter; The delay parameter is obtained by performing average processing on the first delay parameter and the third delay parameter.

[0010] On the other hand, determining the first checkpoint deployment relationship according to the demand parameters of each heterogeneous computing device and the expected time-consuming value of the fault recovery includes: Obtaining a failure probability parameter of each of the heterogeneous computing devices; Sorting the fault probability parameters from large to small to obtain sorted fault probability parameters; Sorting the storage devices corresponding to the expected value of the fault recovery time to obtain the sorted expected value of the fault recovery time; The first checkpoint deployment relationship is determined by performing deployment processing on storage devices corresponding to the sorted failure probability parameters and the sorted expected values ​​of the failure recovery time.

[0011] On the other hand, performing deployment processing on the storage devices corresponding to the sorted failure probability parameters and the sorted expected values ​​of the failure recovery time to determine the first checkpoint deployment relationship includes: Obtain the checkpoint memory of the heterogeneous computing power device corresponding to the sorted failure probability parameters; Selecting the first fault probability parameter from the sorted fault probability parameters as the current fault probability parameter; The storage device corresponding to the last expected value of the fault recovery time after sorting is used as the current storage device; Deploy the checkpoint memory of the heterogeneous computing device corresponding to the current failure probability parameter to the current storage device; The storage space of the storage device corresponding to the expected time of the last failure recovery after the sorting after the deployment is updated; If the storage space of the storage device corresponding to the last expected value of the time required for fault recovery after sorting is greater than or equal to the checkpoint memory of the heterogeneous computing device corresponding to the second failure probability parameter, the second failure probability parameter in the sorted failure probability parameters is used as the new current failure probability parameter, and the process returns to the step of using the storage device corresponding to the last expected value of the time required for fault recovery after sorting as the current storage device; If the storage space of the storage device corresponding to the last expected value of the time taken for fault recovery after the sorting is smaller than the checkpoint memory of the heterogeneous computing device corresponding to the second fault probability parameter, the second fault probability parameter in the sorted fault probability parameters is used as the new current fault probability parameter; the storage device corresponding to the second-to-last expected value of the time taken for fault recovery after the sorting is used as the new current storage device; and the process returns to the step of deploying the checkpoint memory of the heterogeneous computing device corresponding to the current fault probability parameter to the current storage device for deployment until the checkpoint memory of all heterogeneous computing devices is deployed; Create a subset of the deployed heterogeneous computing devices corresponding to each storage device; A mapping relationship is established between each of the subsets and each of the storage devices to determine the first checkpoint deployment relationship.

[0012] On the other hand, the step of optimizing and determining the second checkpoint deployment relationship according to the deployment positions of the heterogeneous computing devices in the first checkpoint deployment relationship includes: The expected time-consuming values ​​of failure recovery corresponding to all storage devices are summed to determine a first optimization target value; Get the initial temperature value and the temperature reduction step; The initial temperature value is used as the current temperature value, and the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship are reassembled to the deployment position of the target storage device to obtain a third checkpoint deployment relationship; The new expected time-consuming values ​​of failure recovery corresponding to all storage devices in the third checkpoint deployment relationship are summed to determine a second optimization target value; When the first optimization target value is greater than the second optimization target value, the third checkpoint deployment relationship is used as a new first checkpoint deployment relationship; the current number of iterations is increased by 1, and the current temperature value is reduced by the temperature reduction step, and the step of recombining the heterogeneous computing devices in the subset corresponding to the storage devices of the first checkpoint deployment relationship to the deployment position of the target storage device to obtain the third checkpoint deployment relationship is returned to the step of recombining the heterogeneous computing devices in the subset corresponding to the storage devices of the first checkpoint deployment relationship to the deployment position of the target storage device to obtain the third checkpoint deployment relationship, until the reduced temperature value or the number of iterations meets the respective corresponding preset conditions, and the iteration is stopped to output the corresponding checkpoint deployment relationship as the second checkpoint deployment relationship; When the first optimization target value is less than or equal to the second optimization target value, obtain the acceptance probability and the random number, and determine the current checkpoint deployment relationship according to the random number and the acceptance probability; add 1 to the current number of iterations, reduce the current temperature value by the temperature reduction step, and return to the step of recombining the heterogeneous computing devices in the subset corresponding to the storage devices of the first checkpoint deployment relationship to the deployment position of the target storage device to obtain the third checkpoint deployment relationship, until the reduced temperature value or the number of iterations meets the corresponding preset conditions, stop the iteration, and output the corresponding checkpoint deployment relationship as the second checkpoint deployment relationship.

[0013] On the other hand, the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship are reassembled to the deployment position of the target storage device to obtain a third checkpoint deployment relationship, including: The corresponding subsets of the heterogeneous computing devices corresponding to the checkpoint deployment in the same storage device are taken as the first set; wherein one storage device corresponds to one first set, and the first set corresponds to one heterogeneous computing device or multiple heterogeneous computing devices; Randomly selecting a target number of first sets in each first set; The heterogeneous computing devices in the first set of the target number are randomly replaced to determine the deployment relationship of the third checkpoint.

[0014] On the other hand, the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship are reassembled to the deployment position of the target storage device to obtain a third checkpoint deployment relationship, including: The corresponding subsets of the heterogeneous computing devices corresponding to the checkpoint deployment in the same storage device are taken as the first set; wherein one storage device corresponds to one first set, and the first set corresponds to one heterogeneous computing device or multiple heterogeneous computing devices; Selecting a target first set including a plurality of heterogeneous computing devices from each first set; The heterogeneous computing devices of the first set where the adjacent target first set is located are randomly replaced to determine the third checkpoint deployment relationship.

[0015] On the other hand, after determining the second checkpoint deployment relationship, when a new training task arrives, it also includes: Get the memory storage path of the new training task; If the memory storage path is occupied by the checkpoint memory, the checkpoint is moved to a storage device corresponding to the minimum latency among the storage devices.

[0016] In order to solve the above technical problems, the present invention also provides a heterogeneous computing system, including a host, a plurality of heterogeneous computing devices and a plurality of storage devices; The host is connected to each heterogeneous computing device respectively; each heterogeneous computing device calls each storage device; The host is used to execute the steps of the above-mentioned checkpoint deployment method for a heterogeneous computing system to complete the checkpoint deployment operation.

[0017] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the checkpoint deployment method for the heterogeneous computing system when executed by a processor.

[0018] In order to solve the above technical problems, the present invention further provides a checkpoint deployment device for a heterogeneous computing system, comprising: Memory for storing computer programs; A processor is used to implement the steps of the checkpoint deployment method for a heterogeneous computing system when executing the computer program.

[0019] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the checkpoint deployment method for a heterogeneous computing system are implemented.

[0020] The beneficial effects of the present invention are that, first, before the checkpoint is placed, the expected time consumption for failure recovery corresponding to each storage device when deploying the checkpoint on each storage device is calculated according to the demand parameters corresponding to each heterogeneous computing power device, so as to weigh the impact of the demand parameters of each heterogeneous computing power device on the checkpoint recovery, so as to achieve the consideration of various factors of the corresponding checkpoint deployment and corresponding recovery speed between the demand parameters of each heterogeneous computing power device. Secondly, the preliminary first checkpoint deployment relationship is determined according to the demand parameters of each heterogeneous computing power device and the calculated expected time consumption for failure recovery of the storage device, and the characteristic parameters such as the demand parameters of each heterogeneous computing power device are combined with the expected time consumption for failure recovery of each storage device, so as to achieve the predictability of the local optimal selection of the deployment of checkpoints to each storage device corresponding to the characteristic parameters of each heterogeneous computing power device during the deployment process. Finally, based on the preliminary first checkpoint deployment relationship, the second checkpoint deployment relationship is determined by optimizing the deployment positions of each heterogeneous computing device, which can effectively jump out of the local optimal solution and increase the possibility of finding the global optimal solution, improve the global search capability, and effectively alleviate the problem of slow recovery of the checkpoint in the deployment process combined with the corresponding demand parameter characteristics between the storage device and the heterogeneous computing device. The checkpoint corresponding to the training task is deployed through the provided second checkpoint deployment relationship to improve the execution efficiency of the training task.

[0021] Secondly, the expected value of the time consumption for fault recovery is determined by multiplying the fault probability and resource parameters to weigh the impact of the fault probability and resource parameters of heterogeneous computing devices on checkpoint recovery, avoiding the low recovery efficiency caused by the conventional deployment scheme not taking into account the setting of resource parameters, so as to improve the efficiency of checkpoint recovery in the future. The process of determining the expected value of the time consumption for fault recovery is used to weigh the impact of the three demand parameters of the fault probability, delay parameter and bandwidth parameter of heterogeneous computing devices on checkpoint recovery, so as to comprehensively consider the demand parameter factors of heterogeneous computing devices in the subsequent checkpoint placement process, and avoid the situation where the checkpoint recovery efficiency decreases due to the conventional fixed position placement. The process of determining the fault probability parameters of each heterogeneous computing device is determined by the weighted sum of the actual fault probability parameters of the historical fault probability parameters and the empirical fault probability parameters, so as to improve the accuracy of the determination of the fault probability parameters. The determination of the read and write delay and the access delay collected, as well as the determination method through performance testing, make the determination process diverse, so as to improve the accuracy and authority of the final determined delay parameters. In the process of determining the first checkpoint deployment relationship, in the subsequent execution process, it is considered that the storage device call with a low expected value of the fault recovery time will be selected to ensure that the subsequent checkpoint recovery speed is faster. Therefore, the heterogeneous computing power device with a high probability of failure is placed in the storage device with a low expected value of the fault recovery time to speed up the subsequent checkpoint recovery speed. The deployment process of the first checkpoint deployment relationship is determined according to the storage device corresponding to the sorted fault probability parameter and the sorted expected value of the fault recovery time, ensuring that the checkpoints of the heterogeneous computing power device with a high probability of failure are stored in the storage device with a low expected value of the fault recovery time to improve the fault point recovery speed. The second checkpoint deployment relationship is obtained by global optimization. Through the probabilistic acceptance criterion, the poor solution is accepted with a certain probability, which helps to jump out of the local optimal solution, increase the possibility of finding the global optimal solution, and improve the computational cost and time of the iteration. The flexible placement of the checkpoints of the training task is achieved through the determined second checkpoint deployment relationship. The third checkpoint deployment relationship is determined by randomly replacing the deployment positions of each heterogeneous computing power device within the first set of the target number. By limiting the number of the first set of the target number, the number of replacements corresponding to the current iteration process is limited, which mainly reflects the randomness of the random replacement and realizes the flexible replacement of the randomness of the checkpoint deployment position under a certain number. The third checkpoint deployment relationship is determined by randomly replacing the deployment positions of the heterogeneous computing power devices in the first set where the adjacent target first set is located. When the failure probabilities corresponding to the heterogeneous computing power devices are relatively small and the expected values ​​of the failure recovery time of the corresponding storage devices are relatively small, replacement is performed to realize the global optimization solution corresponding to the local replacement and realize the diversity and flexibility of the checkpoint deployment positions.If a newly arrived training task occupies the memory of any checkpoint, the storage location of the occupied checkpoint is moved to a storage device with minimal latency and memory space greater than or equal to the checkpoint memory based on the available storage information in the heterogeneous computing system, and the available storage information in the heterogeneous computing system is updated to ensure that the new training task is executed first.

[0022] In addition, the present invention also provides a heterogeneous computing system, product, device and medium, which have the same beneficial effects as the above-mentioned checkpoint deployment method of the heterogeneous computing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a schematic diagram of the architecture of a heterogeneous computing system; Figure 2 A flow chart of a checkpoint deployment method for a heterogeneous computing system provided by an embodiment of the present invention; Figure 3 An architectural diagram of training checkpoint placement for heterogeneous computing systems provided by an embodiment of the present invention; Figure 4 A structural diagram of a checkpoint deployment device for a heterogeneous computing system provided by an embodiment of the present invention; Figure 5 A structural diagram of a checkpoint deployment device for a heterogeneous computing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The core of the present invention is to provide a checkpoint deployment method, system, product and device for a heterogeneous computing system to solve the problem that when a failure occurs in a heterogeneous computing device corresponding to a model training task, the corresponding checkpoint recovery efficiency is low, resulting in reduced execution efficiency of the model training task.

[0027] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0028] Heterogeneous computing mainly refers to the computing method of using computing units with different types of instruction sets and architectures to form a system. Common types of computing units include central processing units (CPU), graphics processing units (GPU), digital signal processors (DSP), field-programmable gate arrays (FPGA), etc. Heterogeneous computing devices are different types of processors and corresponding memories. When processing different types of applications, their performance varies. A heterogeneous computing system is a system that integrates multiple computing units of different types and architectures to improve the level of computing power. In a multi-heterogeneous computing system, heterogeneous computing power (GPU, FPGA and other heterogeneous computing power devices) with different computing performance will be connected to the same distributed computing system for different neural network calculations. In this system, heterogeneous computing power can be allocated to perform neural network calculations that meet its computing characteristics, thereby improving the performance of the calculation.

[0029] Since the heterogeneous computing power of various manufacturers usually has an incomplete software and hardware ecosystem, it is easy to fail or report errors when performing neural network calculations, which brings troubles to the execution of calculations in heterogeneous computing systems. Especially after the emergence of pre-trained models and distributed training requirements, the scale and time of calculations have increased significantly, and the impact of failures on the neural network calculations of heterogeneous computing systems has become more significant. At present, one way is to temporarily store the training checkpoints during the training process. Once the neural network training task fails or is interrupted, the failed heterogeneous computing power can read the training status of the previous model from the stored checkpoints, and then continue training, avoiding the need to retrain a lot of resources and time waste once an error occurs. The conventional method places the checkpoints in a fixed position in the external memory. After a failure occurs, there may be a long loading time due to the low input and output (IO) speed and communication efficiency corresponding to the external memory loading checkpoint, which affects the execution efficiency of the training task.

[0030] Figure 1 It is a schematic diagram of the architecture of a heterogeneous computing system. Figure 1 As shown, in a heterogeneous computing system, the checkpoints of model training are usually backed up to a fixed location in the external memory. The storage IO speed and the slow network interconnection with the server will result in a slow checkpoint recovery efficiency when a heterogeneous computing device fails. The checkpoint deployment method for a heterogeneous computing system provided by the present invention can solve the above problems.

[0031] Figure 2A flowchart of a checkpoint deployment method for a heterogeneous computing system provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes: S11: Obtaining the required parameters of the heterogeneous computing devices and each storage device of the heterogeneous computing system; S12: Deploy a checkpoint for each storage device according to the demand parameters of each heterogeneous computing device to determine the expected time for failure recovery of each storage device; S13: Determine the first checkpoint deployment relationship according to the demand parameters of each heterogeneous computing device and the expected time of failure recovery; S14: Optimize and determine the second checkpoint deployment relationship according to the deployment positions of the heterogeneous computing devices within the first checkpoint deployment relationship, so as to deploy the checkpoints of the target heterogeneous computing devices of the training task according to the second checkpoint deployment relationship.

[0032] Specifically, the number of heterogeneous computing devices in the heterogeneous computing system can be one or more, which is not limited here. The storage device of the heterogeneous computing system in this embodiment can be the local memory of the heterogeneous computing device, or the corresponding external memory, or the sum of the two storage methods. The number of storage devices is currently multiple. The heterogeneous computing device takes into account the deployment scheme setting of each training task before the actual execution in the heterogeneous computing device, and there are multiple heterogeneous computing devices.

[0033] The required parameters of heterogeneous computing power equipment mainly include corresponding performance requirements, network requirements, data transmission requirements, storage requirements and other parameters, so as to achieve the normal operation of heterogeneous computing power equipment in the heterogeneous computing system.

[0034] In step S12, according to the demand parameters of each heterogeneous computing device, the checkpoints corresponding to each heterogeneous computing device are placed on each storage device of the heterogeneous computing system for deployment, and the expected time consumption for failure recovery of the storage device corresponding to the checkpoint when the heterogeneous computing device fails is determined. The expected time consumption for failure recovery is used to weigh the impact of one or more demand parameters of the heterogeneous computing device on the checkpoint recovery. The type of the corresponding demand parameter is not limited here and can be set according to the actual situation.

[0035] The expected value of the fault recovery time is determined by the expectation function. There is no restriction on how to calculate the specific requirement parameters of the corresponding expectation function and what specific expectation function is used. However, based on the definition of the expectation function, the sum of the products of all possible variable values ​​corresponding to it and the failure probability of each heterogeneous computing power device is used. The failure probability of each heterogeneous computing power device is a requirement parameter and can be obtained through a specific algorithm or experience value, or a failure probability value given by the manufacturer, etc., without limitation.

[0036] In step S13, the preliminary first checkpoint deployment relationship is determined according to the demand parameter and the expected time for fault recovery. The checkpoint deployment relationship can exist in the form of a table, a list such as a collection, or a database storage method, which is not limited here. Since the checkpoint deployment relationship is determined by the parameter of the expected time for fault recovery, and at the same time, considering the memory occupied by the storage device, the demand parameter is used as the failure probability of each heterogeneous computing power device. According to the size of the failure probability of each heterogeneous computing power device, the heterogeneous computing power device is prioritized, and in accordance with the principle that the heterogeneous computing power device with a larger failure probability is arranged to the storage device with a smaller expected time for fault recovery, the heterogeneous computing power device with the largest failure probability is preferentially arranged to the storage device with the smallest expected time for fault recovery. It is worth noting that the checkpoints for storing heterogeneous computing power devices on a storage device can be one or more, that is, the number of storage heterogeneous computing power devices corresponding to the checkpoints can be one or more. The main consideration is the available memory in the storage device. If the memory of the current storage device is large enough, the memory of the checkpoint of the corresponding heterogeneous computing device is sufficient to accommodate it, and the memory of the checkpoint with a lower failure probability than that of the heterogeneous computing device can also be placed, then it will be placed in the current storage device.

[0037] It should be noted that the first checkpoint deployment relationship determined in the present embodiment is a local deployment solution considered in response to fault recovery. In order to further achieve a low latency during the deployment process, a global deployment solution is adopted in step S14, and the heterogeneous computing devices of the first checkpoint deployment relationship are deployed in corresponding storage devices and their positions are swapped to optimize and determine the second checkpoint deployment relationship. The optimization algorithm here can be a conventional global optimization algorithm, or it can be based on various factors of checkpoint recovery in the present embodiment to implement a new global optimization algorithm, without limitation.

[0038] The order of deployment positions here can be randomly swapped for the positions of heterogeneous computing devices to which all checkpoints belong, or can be swapped for the positions of heterogeneous computing devices to which checkpoints with the same characteristics belong. After the swap, a new checkpoint deployment relationship is obtained, and the objective function is recalculated. The target value of the objective function is compared with the target value corresponding to the first checkpoint deployment relationship. If the comparison conditions are met, it means that the new checkpoint deployment relationship is better in terms of globality than the preliminarily determined first checkpoint deployment relationship. Therefore, the new checkpoint deployment relationship will be used as the final second checkpoint deployment relationship. Of course, the determination of the checkpoint deployment relationship needs to be achieved through multiple iterations, multiple swaps, etc. to ensure the fairness and authority of the optimization plan.

[0039] After determining the second checkpoint deployment relationship, the checkpoints of the target heterogeneous computing devices of the training task are deployed according to the determined second checkpoint deployment relationship. The training task here corresponds to the actual execution task, and the number of target heterogeneous computing devices corresponding to a training task can be one or more. One heterogeneous computing device corresponds to one training subtask, and one training subtask is used to complete one checkpoint.

[0040] The embodiment of the present invention provides a checkpoint deployment method for a heterogeneous computing system, which obtains the demand parameters of heterogeneous computing devices and each storage device of the heterogeneous computing system; deploys checkpoints corresponding to each storage device according to the demand parameters of each heterogeneous computing device to determine the expected time consumption for failure recovery of each storage device; determines a first checkpoint deployment relationship according to the demand parameters and the expected time consumption for failure recovery of each heterogeneous computing device; optimizes and determines a second checkpoint deployment relationship according to the deployment positions of each heterogeneous computing device within the first checkpoint deployment relationship, so as to facilitate the deployment of checkpoints of the target heterogeneous computing device of the training task according to the second checkpoint deployment relationship. First, before the checkpoint is placed, the expected time consumption for failure recovery corresponding to each storage device when deploying the checkpoint on each storage device is calculated according to the demand parameters corresponding to each heterogeneous computing device, so as to weigh the influence of each demand parameter of each heterogeneous computing device on the checkpoint recovery to achieve the consideration of various factors of the corresponding checkpoint deployment and the corresponding recovery speed between the demand parameters of each heterogeneous computing device. Secondly, the preliminary first checkpoint deployment relationship is determined based on the demand parameters of each heterogeneous computing device and the calculated expected time consumption for failure recovery of the storage device. By combining the characteristic parameters such as the demand parameters of each heterogeneous computing device with the expected time consumption for failure recovery of each storage device, the predictability of the local optimal selection of the checkpoints deployed to each storage device corresponding to the characteristic parameters of each heterogeneous computing device can be achieved during the deployment process. Finally, based on the preliminary first checkpoint deployment relationship, the second checkpoint deployment relationship is determined by optimizing the deployment positions of each heterogeneous computing device, which can effectively jump out of the local optimal solution and increase the possibility of finding the global optimal solution, improve the global search capability, and effectively alleviate the problem of slow recovery of the checkpoints in the deployment process in combination with the corresponding demand parameter characteristics between the storage device and the heterogeneous computing device. The checkpoints corresponding to the training tasks are deployed through the provided second checkpoint deployment relationship to improve the execution efficiency of the training tasks.

[0041] In some embodiments, a checkpoint is deployed for each storage device according to the demand parameters of each heterogeneous computing device to determine the expected time for failure recovery of each storage device, including: Obtain the failure probability parameters of each heterogeneous computing device and the resource parameters of each storage device corresponding to each heterogeneous computing device; According to the failure probability parameters and resource parameters of each heterogeneous computing device, the expected time consumption for failure recovery of each storage device corresponding to each storage device when each heterogeneous computing device deploys a checkpoint is determined.

[0042] Specifically, the failure probability parameter of each heterogeneous computing device refers to the probability of a failure interrupting training when a heterogeneous computing device performs a training task. The resource parameters of each storage device corresponding to each heterogeneous computing device are used to measure the key indicators of computer system or network performance, such as latency parameters, bandwidth parameters and memory space, which jointly affect the overall performance of the system and user experience. The expected time for failure recovery corresponding to the storage device after the checkpoint interval is deployed for each storage device corresponding to each heterogeneous computing device is determined by multiplying the failure probability parameter of each heterogeneous computing device with the resource parameter. Here, all heterogeneous computing devices are calculated for each storage device. For example, there are 5 heterogeneous computing devices and 2 storage devices, and the final expected time for failure recovery is 10.

[0043] The determination of the expected value of the fault recovery time provided in this embodiment is determined by multiplying the fault probability and the resource parameter to weigh the impact of the failure probability and resource parameters of heterogeneous computing devices on checkpoint recovery, thereby avoiding the low recovery efficiency caused by conventional deployment solutions that do not take into account the setting of resource parameters, so as to facilitate the subsequent improvement of checkpoint recovery efficiency.

[0044] In some embodiments, the resource parameters include at least a latency parameter, a bandwidth parameter, and a current memory occupied space of each heterogeneous computing device corresponding to each storage device when each heterogeneous computing device executes a checkpoint corresponding to a training task; determining the expected time consumption for failure recovery of each storage device when each heterogeneous computing device deploys a checkpoint corresponding to each storage device according to the failure probability parameter and resource parameter of each heterogeneous computing device, including: Determine the transmission time based on the current memory usage and bandwidth parameters; Determine the total latency parameter corresponding to each heterogeneous computing device to each storage device based on the transmission time and latency parameter; According to the total delay parameter and the failure probability parameter, the expected value of the failure recovery time corresponding to each storage device when each heterogeneous computing device deploys a checkpoint corresponding to each storage device is determined.

[0045] Specifically, the current memory space occupied by each heterogeneous computing device corresponding to the checkpoint for executing the training task indicates the memory value occupied by the current memory space for subsequent deployment on the storage device. The transmission time is determined by the current memory space occupied and the bandwidth parameter. Specifically, the transmission time is obtained by dividing the current memory space occupied by the bandwidth, which indicates the time required to transmit the checkpoint data of the current memory space occupied under given bandwidth conditions. The total delay parameter corresponding to each heterogeneous computing device to each storage device is determined based on the transmission time and the delay parameter. This parameter comprehensively considers the time for checkpoint data to be transmitted in the network (transmission time) and the time for processing or waiting in the network node (delay parameter). The expected value of the fault recovery time corresponding to each storage device is obtained by multiplying the total delay parameter and the failure probability parameter. The specific formula is as follows: ; in, is the i-th heterogeneous computing device, is the storage device corresponding to the i-th heterogeneous computing device, is the latency parameter from the ith heterogeneous computing device to the corresponding storage device, is the bandwidth parameter from the ith heterogeneous computing device to the corresponding storage device, The current memory usage of the checkpoint corresponding to the training task executed by the i-th heterogeneous computing device; is the failure probability parameter of the i-th heterogeneous computing device.

[0046] The process for determining the expected value of the fault recovery time provided in this embodiment is used to weigh the impact of the three demand parameters of the heterogeneous computing power equipment, namely the failure probability, delay parameter and bandwidth parameter, on the checkpoint recovery, so as to comprehensively consider the demand parameter factors of the heterogeneous computing power equipment in the subsequent checkpoint placement process, and avoid the situation where the checkpoint recovery efficiency is reduced due to the conventional fixed-position placement.

[0047] In some embodiments, the process of determining the failure probability parameters of each heterogeneous computing device includes: Obtain the empirical failure probability parameters of the current heterogeneous computing power equipment; Determine the historical failure probability parameters of the current heterogeneous computing power equipment within a preset time interval; Determine a corresponding first weight parameter according to the amount of maintenance data of the historical failure probability parameter; Determine a second weight parameter of the empirical fault probability parameter according to the first weight parameter; The failure probability parameter of the current heterogeneous computing power device is determined based on the historical failure probability parameter, the first weight parameter, the empirical failure probability parameter and the second weight parameter.

[0048] Specifically, the fault probability parameter of this embodiment is based on the existing fault analysis mechanism, and the characteristics of the fault are clearly reflected in the specific performance parameters. It is not limited here. As long as the fault probability parameter can be obtained, the specific fault type is not limited, and it can be a hardware fault or a software fault.

[0049] The empirical fault probability parameters can be obtained through conventional empirical values, or by searching existing materials, or by being determined by the manufacturer. The historical fault probability parameters correspond to the actual historical fault probability parameters currently collected within the preset time interval. The setting of the empirical fault probability parameters and the historical fault probability parameters can determine the weight parameters through the probability model, or the weight of the empirical fault probability parameters can be affected by checking the amount of maintenance data of the historical fault probability parameters, so as to determine the corresponding first weight parameter setting and determine the second weight parameter (1-first weight parameter) through the amount of maintenance data of the historical fault probability parameters. The current failure probability parameter of the heterogeneous computing power equipment = empirical failure probability parameter * first weight parameter + historical failure probability parameter * second weight parameter.

[0050] The process of determining the failure probability parameters of each heterogeneous computing device provided in this embodiment is determined by weighted summation of actual failure probability parameters of historical failure probability parameters and empirical failure probability parameters, so as to improve the accuracy of determining the failure probability parameters.

[0051] In some embodiments, the process of determining the latency parameter of each storage device corresponding to each heterogeneous computing device includes: Obtain the read / write latency and access latency from the current heterogeneous computing device to each storage device; Determine a first delay parameter according to the read / write delay and the access delay; Obtain the query rate of the current heterogeneous computing device under each concurrent condition to determine the corresponding second delay parameter; The third time delay parameter is obtained by performing average processing on the second time delay parameters; The first delay parameter and the third delay parameter are averaged to obtain the delay parameter.

[0052] It should be noted that read and write latency and access latency are two important performance parameters that jointly determine the latency parameters of each storage device corresponding to each heterogeneous computing device. Read and write latency is the time required from initiating a read and write operation to the completion of the operation, including the time it takes for data to be read from the storage medium of the current heterogeneous computing device to the memory, or from the memory to be written to the storage medium. Access latency is the time required to access checkpoint data from the storage device, which is usually affected by the memory speed and hit rate of the storage device. The first latency parameter is determined based on the read and write latency and the access latency.

[0053] In addition, in order to improve the accuracy of obtaining the delay parameter, the second delay parameter can be determined by comparing the query rates per second (QPS) of different storage media under different concurrency conditions. The third delay parameter is obtained by averaging the second delay parameters. The delay parameter is obtained by averaging the first delay parameter and the third delay parameter.

[0054] It should be noted that the first delay parameter and the second delay parameter can be independently used as technical solutions for the current delay parameter determination process, and are not limited here.

[0055] This embodiment provides the determination of the collected read and write delays and access delays, and the determination method through performance testing, so as to diversify the determination process and improve the accuracy and authority of the delay parameters finally determined.

[0056] In some embodiments, determining the first checkpoint deployment relationship according to the demand parameters of each heterogeneous computing device and the expected time for failure recovery includes: Obtain the failure probability parameters of each heterogeneous computing device; Sort each fault probability parameter from large to small to obtain sorted fault probability parameters; Sorting the storage devices corresponding to the expected values ​​of the time taken for failure recovery to obtain sorted expected values ​​of the time taken for failure recovery; The first checkpoint deployment relationship is determined by performing deployment processing on storage devices corresponding to the sorted failure probability parameters and the sorted expected failure recovery time values.

[0057] Specifically, the determination of the first checkpoint deployment relationship is based on the principle of sorting the failure probability parameters from large to small, and preferentially deploying heterogeneous computing devices with larger failure probability parameters to storage devices with smaller expected failure recovery time to determine the first checkpoint deployment relationship. At this time, the storage devices corresponding to the expected failure recovery time need to be sorted to increase the speed of deployment processing.

[0058] The process of determining the first checkpoint deployment relationship provided in this embodiment takes into account the selection of storage devices with a low expected time for failure recovery during subsequent execution to ensure that the subsequent checkpoint recovery speed is fast. Therefore, heterogeneous computing devices with a high probability of failure are placed in storage devices with a low expected time for failure recovery to speed up the subsequent checkpoint recovery speed.

[0059] In some embodiments, performing deployment processing on storage devices corresponding to the sorted failure probability parameters and the sorted expected failure recovery time values ​​to determine the first checkpoint deployment relationship includes: Obtain the checkpoint memory of the heterogeneous computing power device corresponding to the sorted failure probability parameters; Selecting the first fault probability parameter from the sorted fault probability parameters as the current fault probability parameter; The storage device corresponding to the last expected value of the fault recovery time after sorting is used as the current storage device; Deploy the checkpoint memory of the heterogeneous computing device corresponding to the current failure probability parameter to the current storage device; The storage space of the storage device corresponding to the expected time of the last failure recovery after the sorting after the deployment is updated; If the storage space of the storage device corresponding to the last expected value of the time required for fault recovery after sorting is greater than or equal to the checkpoint memory of the heterogeneous computing device corresponding to the second failure probability parameter, the second failure probability parameter in the sorted failure probability parameters is used as the new current failure probability parameter, and the process returns to the step of using the storage device corresponding to the last expected value of the time required for fault recovery after sorting as the current storage device; If the storage space of the storage device corresponding to the last expected value of the fault recovery time after sorting is smaller than the checkpoint memory of the heterogeneous computing power device corresponding to the second fault probability parameter, the second fault probability parameter in the sorted fault probability parameters is used as the new current fault probability parameter; the storage device corresponding to the second-to-last expected value of the fault recovery time after sorting is used as the new current storage device; and the process returns to the step of deploying the checkpoint memory of the heterogeneous computing power device corresponding to the current fault probability parameter to the current storage device for deployment until the checkpoint memory of all heterogeneous computing power devices is deployed; Create a subset of the deployed heterogeneous computing devices corresponding to each storage device; A mapping relationship is established between each subset and each storage device to determine a first checkpoint deployment relationship.

[0060] Specifically, the sorted fault probability parameters are arranged from the largest to the smallest, and the first fault probability parameter, that is, the largest fault probability parameter, is selected as the current fault probability parameter. At this time, the storage device corresponding to the last expected value of fault recovery time (the smallest expected value of fault recovery time) after sorting is used as the current storage device, and the checkpoint memory of the heterogeneous computing power device corresponding to the current fault probability parameter is deployed to the current storage device, and the storage space of the current storage device is updated at the same time. If the storage space of the updated current storage device is greater than or equal to the checkpoint memory of the heterogeneous computing power device corresponding to the second fault probability parameter, it means that the storage space of the current storage device is sufficient to add the checkpoint memory corresponding to the second fault probability parameter, so it can continue to be added. If the storage space of the storage device corresponding to the last expected value of fault recovery time after sorting is less than the checkpoint memory of the heterogeneous computing power device corresponding to the second fault probability parameter, it means that the storage space of the current storage device is insufficient to add the checkpoint memory corresponding to the second fault probability parameter, and a new current storage device needs to be obtained. Here, the new current storage device is the storage device with the second to last expected value of fault recovery time, and this order is followed until the checkpoint memory of the heterogeneous computing power device is deployed.

[0061] It should be noted that the memory of all storage devices is larger than the memory of the corresponding checkpoints of all heterogeneous computing devices, so that all heterogeneous computing devices can meet the requirement of having storage device storage checkpoints. Therefore, there may be a deployment situation where there is no checkpoint corresponding to the heterogeneous computing device in a certain storage device. Therefore, corresponding subsets are established for the deployed heterogeneous computing devices corresponding to each storage device, and the mapping relationship between the corresponding subsets established in a storage device determines the first checkpoint deployment relationship.

[0062] This embodiment provides a deployment process for determining the first checkpoint deployment relationship by deploying storage devices corresponding to sorted failure probability parameters and sorted failure recovery time expected values, thereby ensuring that checkpoints of heterogeneous computing devices with higher failure probabilities are stored in storage devices with lower failure recovery time expected values, thereby improving the failure point recovery speed.

[0063] In some embodiments, optimizing and determining the second checkpoint deployment relationship according to the deployment positions of the heterogeneous computing devices within the first checkpoint deployment relationship includes: The expected time-consuming values ​​of failure recovery corresponding to all storage devices are summed to determine a first optimization target value; Get the initial temperature value and the temperature reduction step; The initial temperature value is used as the current temperature value, and the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship are reassembled to the deployment position of the target storage device to obtain a third checkpoint deployment relationship; The new expected time-consuming values ​​of failure recovery corresponding to all storage devices in the third checkpoint deployment relationship are summed to determine a second optimization target value; When the first optimization target value is greater than the second optimization target value, the third checkpoint deployment relationship is used as the new first checkpoint deployment relationship; the current number of iterations is increased by 1, and the current temperature value is reduced by the temperature reduction step, and the process returns to the step of recombining the heterogeneous computing devices in the subset corresponding to the storage devices of the first checkpoint deployment relationship to the deployment position of the target storage device to obtain the third checkpoint deployment relationship, until the reduced temperature value or the number of iterations meets the corresponding preset conditions, and the iteration is stopped to output the corresponding checkpoint deployment relationship as the second checkpoint deployment relationship; When the first optimization target value is less than or equal to the second optimization target value, the acceptance probability is obtained, and a random number is obtained, and the current checkpoint deployment relationship is determined according to the random number and the acceptance probability; and the current number of iterations is increased by 1, and the current temperature value is reduced by the temperature reduction step, and the process returns to the step of recombining the heterogeneous computing devices in the subset corresponding to the storage devices of the first checkpoint deployment relationship to the deployment position of the target storage device to obtain the third checkpoint deployment relationship, until the reduced temperature value or the number of iterations meets the corresponding preset conditions, and the iteration is stopped to output the corresponding checkpoint deployment relationship as the second checkpoint deployment relationship.

[0064] Specifically, the expected time-consuming values ​​of failure recovery corresponding to each storage device corresponding to the first checkpoint deployment relationship are summed to determine the first optimization target value, and the formula is as follows: ; Where N represents the total number of heterogeneous computing devices, 1≤i≤N; R is the sum of the expected values ​​of the fault recovery time of all heterogeneous computing devices corresponding to all storage devices, that is, the first optimization target value; is the i-th heterogeneous computing device, is the storage device corresponding to the i-th heterogeneous computing device, is the latency parameter from the ith heterogeneous computing device to the corresponding storage device, is the bandwidth parameter from the ith heterogeneous computing device to the corresponding storage device, The current memory usage of the checkpoint corresponding to the training task executed by the i-th heterogeneous computing device; is the failure probability parameter of the i-th heterogeneous computing device.

[0065] The initial temperature value is T, and the temperature reduction step is z. At the initial temperature value, the heterogeneous computing devices in the subset corresponding to each storage device of the first checkpoint deployment relationship are recombined to the deployment position of the target storage device to obtain the third checkpoint deployment relationship. The recombination here can be a random swap of heterogeneous computing devices in any multiple sets to generate a new subset corresponding to the storage device, provided that the memory of the storage device can carry the checkpoint memory corresponding to each heterogeneous computing device. The random swap here can correspond to the swap of the checkpoint containing only one heterogeneous computing device in the subset, or it can correspond to the swap of the checkpoint containing multiple heterogeneous computing devices in the subset, etc., which is not limited here.

[0066] The new expected values ​​of fault recovery time corresponding to all storage devices in the third checkpoint deployment relationship are summed to obtain the second optimization target value. The process of obtaining the new expected value of fault recovery time here is the same as the process of obtaining the original expected value of fault recovery time, which will not be repeated here.

[0067] The first optimization target value is compared with the second optimization target value. If the first optimization target value is greater than the second optimization target value, it means that the original first checkpoint deployment relationship is better optimized. At this time, it is necessary to perform the next method to replace it according to the number of iterations and the temperature reduction to obtain a new third checkpoint deployment relationship. The comparison is performed until the reduced temperature value or the number of iterations meets the corresponding preset conditions. The iteration is stopped to output the checkpoint deployment relationship corresponding to the optimization as the second checkpoint deployment relationship.

[0068] If the first optimization target value is less than or equal to the second optimization target value, the acceptance probability is obtained, a random number in the interval (0, 1) is generated, and the current checkpoint deployment relationship is determined based on the comparison between the random number and the acceptance probability. At the same time, the iteration method is the same as the previous method, which will not be repeated here. Until the second checkpoint deployment relationship is determined.

[0069] The global optimization provided in this embodiment obtains the second checkpoint deployment relationship, and accepts the poor solution with a certain probability through the probabilistic acceptance criterion, thereby helping to jump out of the local optimal solution, increase the possibility of finding the global optimal solution, and improve the computational cost and time of the iteration. The flexible placement of the checkpoints of the training task is achieved through the determined second checkpoint deployment relationship.

[0070] In some embodiments, determining the current checkpoint deployment relationship according to the random number and the acceptance probability includes: If the acceptance probability is greater than the random number, the third checkpoint deployment relationship is used as the new first checkpoint deployment relationship; If the acceptance probability is less than or equal to the random number, the first checkpoint deployment relationship is retained; The process of determining the acceptance probability includes: Performing difference processing on the second optimization target value and the first optimization target value to obtain an optimization target difference; The optimization target difference is divided by the current temperature value to obtain a first index to determine the acceptance probability.

[0071] Specifically, when the acceptance probability is greater than the random number, the third checkpoint deployment relationship is used as the new first checkpoint deployment relationship. When the acceptance probability is less than or equal to the random number, the first checkpoint deployment relationship is retained.

[0072] At this time, the acceptance probability is the difference between the second optimization target value and the first optimization target value to obtain the optimization target difference. The optimization target difference is divided by the current temperature value to obtain the first index to determine the acceptance probability. The specific formula is as follows: Probability of acceptance ; Among them, R new is the second optimization target value, R is the first optimization target value, and T is the current temperature value.

[0073] The present embodiment provides a method for determining a probabilistic acceptance criterion to speed up the process of obtaining the global optimal solution, thereby reducing the iteration cost of determining the second checkpoint deployment relationship, and simplifying the operation steps of parameter iteration, which is easy to implement.

[0074] In some embodiments, the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship are reassembled to the deployment position of the target storage device to obtain a third checkpoint deployment relationship, including: The corresponding subsets of the heterogeneous computing devices corresponding to the checkpoint deployment in the same storage device are taken as the first set; wherein one storage device corresponds to one first set, and the first set corresponds to one heterogeneous computing device or multiple heterogeneous computing devices; Randomly selecting a target number of first sets in each first set; The heterogeneous computing devices in the first set of the target number are randomly replaced to determine the deployment relationship of the third checkpoint.

[0075] Specifically, the subsets corresponding to the heterogeneous computing devices in the same storage device are taken as the first set, and the target number of first sets are selected in each first set, and the deployment positions corresponding to the heterogeneous computing devices in the target number of first sets are randomly replaced. The target number of first sets here can correspond to several sets, and can be randomly selected or determined by the storage number of the checkpoints of the corresponding heterogeneous computing devices in each set, etc., which are not limited here. There is no limitation on random replacement, and the deployment position of the checkpoints of only one heterogeneous computing device can be changed, or the deployment positions of the checkpoints of multiple heterogeneous computing devices can be changed.

[0076] The third checkpoint deployment relationship determined by randomly replacing the deployment positions of each heterogeneous computing power device within the first set of the target number provided in this embodiment limits the number of replacements corresponding to the current iteration process by limiting the number of the first set of the target number, which mainly reflects the randomness of the random replacement and realizes the flexible replacement of the randomness of the checkpoint deployment position under a certain number.

[0077] In some other embodiments, the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship are reassembled to the deployment position of the target storage device to obtain a third checkpoint deployment relationship, including: The corresponding subsets of the heterogeneous computing devices corresponding to the checkpoint deployment in the same storage device are taken as the first set; wherein one storage device corresponds to one first set, and the first set corresponds to one heterogeneous computing device or multiple heterogeneous computing devices; Selecting a target first set including a plurality of heterogeneous computing devices from each first set; The heterogeneous computing devices of the first set where the adjacent target first set is located are randomly replaced to determine the third checkpoint deployment relationship.

[0078] Specifically, this embodiment takes into account the checkpoint deployment situation corresponding to multiple heterogeneous computing devices in a set and uses it as the target first set. Since the storage devices corresponding to the first checkpoint deployment relationship are sorted accordingly, the adjacent target first set is selected at this time, and the failure probabilities of the heterogeneous computing devices are relatively small, and the expected values ​​of the corresponding storage devices for failure recovery are relatively small, and they are replaced. Based on the small difference, the corresponding third checkpoint deployment relationship is obtained to achieve local improvement under global optimization.

[0079] The third checkpoint deployment relationship determined by randomly replacing the deployment positions of heterogeneous computing devices in the first set of adjacent target first sets provided in this embodiment is replaced when the failure probabilities corresponding to the various heterogeneous computing devices are slightly different and the expected values ​​of the failure recovery times of the corresponding storage devices are slightly different, thereby realizing a global optimization solution corresponding to the local replacement and achieving diversity and flexibility of the checkpoint deployment positions.

[0080] In some embodiments, after determining the second checkpoint deployment relationship, when a new training task arrives, the method further includes: Get the memory storage path of the new training task; If the memory storage path is occupied by the checkpoint memory, the checkpoint is moved to the storage device with the minimum latency and a memory space greater than or equal to the checkpoint memory.

[0081] Specifically, after determining the final second checkpoint deployment relationship, if the memory storage path corresponding to the checkpoint is occupied, but the memory storage path corresponding to the new training task is the same as the memory storage path corresponding to the checkpoint, the execution of the new training task is currently prioritized, and the checkpoint is migrated to the storage device with the corresponding minimum latency and memory space greater than or equal to the checkpoint memory, so as to ensure the smooth execution of the new training task.

[0082] This embodiment provides that if a newly arrived training task occupies the memory of any checkpoint, the storage location of the occupied checkpoint is moved to a storage device with minimal latency and memory space greater than or equal to the checkpoint memory based on the available storage information in the heterogeneous computing system, and the available storage information in the heterogeneous computing system is updated to ensure that the new training task is executed first.

[0083] Figure 3 An architecture diagram of a training checkpoint placement for a heterogeneous computing system provided by an embodiment of the present invention, such as Figure 3 As shown, it includes a training task information collection module, a heterogeneous computing system storage information collection module, a heterogeneous computing power equipment failure probability statistics module and a checkpoint position adjustment module.

[0084] 1. Training task information collection module: When a user issues a training task, this module is used to collect the information of the training task and notify the heterogeneous computing system storage information collection module and the heterogeneous computing power equipment failure probability statistics module to perform corresponding information collection and statistics. The information of the training task will be sent to the checkpoint position adjustment module.

[0085] 2. Heterogeneous computing system storage information collection module: After receiving the notification from the training task information collection module, this module will collect the available storage information in the heterogeneous computing system and send it to the checkpoint location adjustment module.

[0086] 3. Heterogeneous computing power equipment failure probability statistics module: After receiving the notification from the training task information collection module, this module will collect the failure probability information of the heterogeneous computing power equipment related to the training task information and send it to the checkpoint position adjustment module.

[0087] 4. Checkpoint location adjustment module: Based on the information from the above three modules, this module will calculate the checkpoint placement location for the training task and send the checkpoint placement location information to the heterogeneous computing system for actual execution.

[0088] The following is the workflow corresponding to each module: 1. Training task information collection module: When a user submits a new distributed training task, the training task information collection module will collect information about the training task, including: a. Heterogeneous computing devices corresponding to training tasks. It is worth noting that a training task may correspond to one heterogeneous computing device or multiple heterogeneous computing devices, depending on whether the training task is single-card training or multi-card training.

[0089] b. How much memory space is occupied by the model assigned to the training task on the relevant heterogeneous computing devices. This memory space can be calculated based on the model assigned to the training task on each heterogeneous computing device. The calculation can be performed directly using tools or by writing code yourself.

[0090] The training task information will be sent to the checkpoint location adjustment module. At the same time, the training task information collection module will also notify the heterogeneous computing system storage information collection module and the heterogeneous computing power equipment failure probability statistics module to start working and perform corresponding storage information collection and failure probability statistics.

[0091] 2. Heterogeneous computing system storage information collection module: Upon receiving notification from the training task information collection module, the heterogeneous computing system storage information collection module collects available storage information in the current heterogeneous computing system, including: a. The available storage space of all storage nodes in the heterogeneous computing system, including the memory, CPU memory, SSD and HDD storage available space of each heterogeneous computing device. The available storage space does not include the memory space required for the training task itself and the occupied storage space.

[0092] b. The latency and bandwidth from the heterogeneous computing devices corresponding to the training task to all storage nodes (storage devices). This indicator can be obtained through the storage performance testing tool and stored in a matrix to save memory resources.

[0093] The available storage information in the heterogeneous computing system is sent to the checkpoint location adjustment module.

[0094] 3. Heterogeneous computing equipment failure probability statistics module: When receiving the notification from the training task information collection module, the heterogeneous computing power device failure probability statistics module will collect the failure probability information of the heterogeneous computing power devices related to the training task information. The failure probability refers to the probability of a failure interrupting the training after a heterogeneous computing power executes a training task. The data can be initialized based on the failure probability of each heterogeneous computing power device queried online, and then weighted updated based on the actual failure occurrence of each corresponding heterogeneous computing power device in the heterogeneous computing within a period of time. For example, the failure probability of heterogeneous computing power device A can be: obtain the empirical failure probability of heterogeneous computing power device A * 0.5 + the actual failure probability of heterogeneous computing power device A in the heterogeneous computing system within one month * 0.5.

[0095] This failure probability information will be sent to the checkpoint location adjustment module.

[0096] 4. Checkpoint position adjustment module: This module records the storage location of previous checkpoints. Based on the storage location of previous checkpoints, the information of received training tasks, and the available storage information in the heterogeneous computing system, if the newly arrived training task occupies the memory of any checkpoint, the storage location of the occupied checkpoint is moved to an available storage device with the minimum latency and memory space greater than or equal to the checkpoint memory, and the available storage information in the heterogeneous computing system is updated.

[0097] Subsequently, based on the information of the training task, the available storage information (after update) in the heterogeneous computing system, and the failure probability information, the module executes the training checkpoint placement algorithm for the heterogeneous computing system to obtain the optimal checkpoint placement method for the training task.

[0098] This module will allocate the storage location of the checkpoint for the training task based on the calculation results, and send it to the actual heterogeneous computing system, while recording the storage location of the checkpoint.

[0099] If any training task ends, the heterogeneous computing system will inform the checkpoint location adjustment module to release the related checkpoint storage and update the storage location of the checkpoint recorded in the module.

[0100] Specifically, according to the information of the training task, the available storage information in the heterogeneous computing system, and the failure probability information, a better checkpoint location is assigned to the newly arrived training task to reduce the impact of the failure on the training task. The algorithm steps are as follows: Step 1: According to the collected information, the latency and bandwidth from the heterogeneous computing device C to the storage device S are , , the memory usage of the trained model in heterogeneous computing power is , the failure probability of computing power is , for all relevant heterogeneous computing devices, the expected time function for failure recovery when placing a checkpoint on any storage device is calculated as: ; in, For heterogeneous computing devices, For storage devices, is the latency parameter from heterogeneous computing devices to storage devices, is the bandwidth parameter from heterogeneous computing devices to storage devices, The current memory usage of the checkpoint corresponding to the training task executed by the heterogeneous computing power device; is the failure probability parameter of heterogeneous computing devices.

[0101] This function can weigh the impact of failure probability, latency, and bandwidth of heterogeneous computing devices on checkpoint recovery.

[0102] Step 2: Sort the related heterogeneous computing devices by the probability of failure from high to low, starting with the heterogeneous computing device with the highest probability of failure, and place their checkpoints one by one. Minimum storage device On, at the same time The available memory is reduced During this process, if a checkpoint cannot be placed on the storage device due to available memory limitations, , then try to place the checkpoint at On a higher storage device, and so on. Finally, a checkpoint placement list of the initial result of the checkpoint placement of a storage device can be obtained. , where set represents a set of one or more heterogeneous computing devices, such as {C1}, or {C2, C3}, Represents placing the checkpoints of the heterogeneous computing devices in the set to the storage device superior.

[0103] Step 3: For checkpoint placement list , further optimize the checkpoint placement, that is, define the overall optimization goal as minimizing R: ; Wherein, N represents the total number of heterogeneous computing power devices, and 1 ≤ i ≤ N; R is the sum of the expected failure recovery time of all heterogeneous computing power devices corresponding to all storage devices, that is, the first optimization target value; is the i-th heterogeneous computing power device, is the storage device corresponding to the i-th heterogeneous computing power device, is the latency parameter from the i-th heterogeneous computing power device to the corresponding storage device, is the bandwidth parameter from the i-th heterogeneous computing power device to the corresponding storage device, is the current memory occupancy of the checkpoint corresponding to the training task executed by the i-th heterogeneous computing power device; is the failure probability parameter of the i-th heterogeneous computing power device.

[0104] Given the initial temperature value T, the temperature reduction step size z, and the number of iterations K, the parameters can be set by oneself.

[0105] Set the checkpoint placement list in step 2 as the current solution, and the initial optimization target value R is obtained under the condition of this current solution, denoted as R old .

[0106] Each time, select 2 sets from the current solution and try to recombine the checkpoint placement of the heterogeneous computing power devices in the two selected sets on the premise that the set is not empty and the device memory is sufficient to carry the checkpoint. If there is no new available combination, then these two sets are considered unavailable and two new sets are selected. If there are no available two sets, then exit the algorithm and directly output the initial checkpoint placement list .

[0107] Calculate R corresponding to the recombined checkpoint placement list, denoted as R new . If R new < R, then use the recombined checkpoint placement list as the current solution. If R new >= R, then with probability use the recombined checkpoint placement list as the current solution.

[0108] Reduce the temperature T by the step size z. If the temperature < 0 or the number of iterations K is satisfied, then terminate the iteration and output the checkpoint placement list of the current solution . Otherwise, return to the technical solution corresponding to selecting 2 sets from the current solution each time in step 3.

[0109] Step 4: Output as the final solution, that is, deploy the checkpoint data of the heterogeneous computing power devices in on S.

[0110] Furthermore, the present invention also provides a heterogeneous computing system, comprising a host, a plurality of heterogeneous computing devices and a plurality of storage devices; The host is connected to each heterogeneous computing device respectively; each heterogeneous computing device calls each storage device; The host is used to execute the steps of the above-mentioned checkpoint deployment method for a heterogeneous computing system to complete the checkpoint deployment operation.

[0111] For an introduction to a heterogeneous computing system provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above checkpoint deployment method for the heterogeneous computing system.

[0112] Furthermore, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the checkpoint deployment method for a heterogeneous computing system when executed by a processor.

[0113] For an introduction to a computer program product provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above checkpoint deployment method for heterogeneous computing systems.

[0114] The above describes in detail various embodiments corresponding to the checkpoint deployment method for a heterogeneous computing system. On this basis, the present invention also discloses a checkpoint deployment device for a heterogeneous computing system corresponding to the above method. Figure 4 A structural diagram of a checkpoint deployment device for a heterogeneous computing system provided by an embodiment of the present invention. Figure 4 As shown, the checkpoint deployment device of the heterogeneous computing system includes: An acquisition module 11 is used to acquire the demand parameters of the heterogeneous computing devices and each storage device of the heterogeneous computing system; A first determination module 12 is used to deploy a checkpoint corresponding to each storage device according to the demand parameters of each heterogeneous computing device to determine the expected time consumption for failure recovery of each storage device; The second determination module 13 is used to determine the first checkpoint deployment relationship according to the demand parameters of each heterogeneous computing device and the expected time value of failure recovery; The third determination module 14 is used to optimize and determine the second checkpoint deployment relationship according to the deployment positions of each heterogeneous computing power device within the first checkpoint deployment relationship, so as to deploy the checkpoints of the target heterogeneous computing power device of the training task according to the second checkpoint deployment relationship.

[0115] Since the embodiments of the device part correspond to the above embodiments, please refer to the description of the embodiments of the method part for the embodiments of the device part, and will not be repeated here.

[0116] For an introduction to a checkpoint deployment device for a heterogeneous computing system provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above checkpoint deployment method for a heterogeneous computing system.

[0117] Figure 5 A structural diagram of a checkpoint deployment device for a heterogeneous computing system provided by an embodiment of the present invention, such as Figure 5 As shown, the device comprises: A memory 21, used for storing computer programs; The processor 22 is configured to implement the steps of the checkpoint deployment method for a heterogeneous computing system when executing a computer program.

[0118] The checkpoint deployment device for the heterogeneous computing system provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0119] Among them, the processor 22 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 22 can be implemented in at least one hardware form of a digital signal processor (DSP), an FPGA, and a programmable logic array (PLA). The processor 22 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU; the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 22 may be integrated with a Graphics Processing Unit, GPU, which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 22 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0120] The memory 21 may include one or more computer-readable storage media, which may be non-transitory. The memory 21 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 21 is at least used to store the following computer program 211, wherein, after the computer program is loaded and executed by the processor 22, it can implement the relevant steps of the checkpoint deployment method for the heterogeneous computing system disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 21 may also include an operating system 212 and data 213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 212 may include Windows, Unix, Linux, etc. The data 213 may include, but is not limited to, data involved in the checkpoint deployment method for the heterogeneous computing system, etc.

[0121] In some embodiments, the checkpoint deployment device of the heterogeneous computing system may further include a display screen 23 , an input and output interface 24 , a communication interface 25 , a power supply 26 , and a communication bus 27 .

[0122] Those skilled in the art can understand that Figure 5 The structure shown in the figure does not constitute a limitation on the checkpoint deployment device of the heterogeneous computing system, and may include more or less components than those shown in the figure.

[0123] The processor 22 implements the checkpoint deployment method for a heterogeneous computing system provided by any of the above embodiments by calling instructions stored in the memory 21 .

[0124] For an introduction to a checkpoint deployment device for a heterogeneous computing system provided by the present invention, please refer to the above method embodiment, and the present invention will not be repeated here. It has the same beneficial effects as the above checkpoint deployment method for a heterogeneous computing system.

[0125] Furthermore, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 22, the steps of the checkpoint deployment method for the heterogeneous computing system as described above are implemented.

[0126] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0127] For an introduction to a computer-readable storage medium provided by the present invention, please refer to the above method embodiment, which will not be described in detail herein. It has the same beneficial effects as the above checkpoint deployment method for heterogeneous computing systems.

[0128] The above is a detailed introduction to the checkpoint deployment method, system, product and device for a heterogeneous computing system provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0129] It should also be noted that, in this specification, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A checkpoint deployment method for a heterogeneous computing system, characterized in that: include: Obtain the required parameters of heterogeneous computing devices and storage devices of heterogeneous computing systems; Deploy a checkpoint for each storage device according to the demand parameters of each heterogeneous computing device to determine the expected time for failure recovery of each storage device; Determine the first checkpoint deployment relationship according to the demand parameters of each heterogeneous computing device and the expected time consumption of the fault recovery; The second checkpoint deployment relationship is determined by optimizing the deployment positions of the heterogeneous computing devices within the first checkpoint deployment relationship, so as to deploy the checkpoints of the target heterogeneous computing devices of the training task according to the second checkpoint deployment relationship.

2. The checkpoint deployment method for a heterogeneous computing system according to claim 1, characterized in that: Deploy a checkpoint for each storage device according to the demand parameters of each heterogeneous computing device to determine the expected time for failure recovery of each storage device, including: Obtaining a failure probability parameter of each heterogeneous computing device and a resource parameter of each storage device corresponding to each heterogeneous computing device; The expected time consumption for failure recovery corresponding to each storage device when each storage device corresponding to each heterogeneous computing device deploys the checkpoint is determined according to the failure probability parameter of each heterogeneous computing device and the resource parameter.

3. The checkpoint deployment method for a heterogeneous computing system according to claim 2, characterized in that: The resource parameters at least include a latency parameter, a bandwidth parameter, and a current memory occupied space of each heterogeneous computing device corresponding to each storage device when each heterogeneous computing device executes a checkpoint corresponding to a training task; determining the expected time consumption for failure recovery corresponding to each storage device when each heterogeneous computing device deploys the checkpoint corresponding to each storage device according to the failure probability parameter of each heterogeneous computing device and the resource parameters, including: Determine the transmission time according to the current memory occupied space and the bandwidth parameter; Determine the total delay parameter corresponding to each of the heterogeneous computing devices to each storage device according to the transmission time and delay parameter; The expected time consumption for failure recovery corresponding to each storage device when each storage device corresponding to each heterogeneous computing device deploys the checkpoint is determined according to the total delay parameter and the failure probability parameter.

4. The checkpoint deployment method for a heterogeneous computing system according to claim 2 or 3, characterized in that: The process of determining the failure probability parameters of each of the heterogeneous computing devices includes: Obtain the empirical failure probability parameters of the current heterogeneous computing power equipment; Determine the historical failure probability parameters of the current heterogeneous computing power equipment within a preset time interval; Determine a corresponding first weight parameter according to the amount of maintenance data of the historical fault probability parameter; Determine a second weight parameter of the empirical fault probability parameter according to the first weight parameter; The failure probability parameter of the current heterogeneous computing power device is determined according to the historical failure probability parameter, the first weight parameter, the empirical failure probability parameter and the second weight parameter.

5. The checkpoint deployment method for a heterogeneous computing system according to claim 3, characterized in that: The process of determining the latency parameter of each storage device corresponding to each of the heterogeneous computing devices includes: Obtain the read / write latency and access latency from the current heterogeneous computing device to each storage device; Determine a first delay parameter according to the read / write delay and the access delay; Obtain the query rate of the current heterogeneous computing device under each concurrent condition to determine the corresponding second delay parameter; Taking an average value of each of the second delay parameters to obtain a third delay parameter; The delay parameter is obtained by performing average processing on the first delay parameter and the third delay parameter.

6. The checkpoint deployment method for a heterogeneous computing system according to claim 2, characterized in that: The determining of the first checkpoint deployment relationship according to the demand parameters of each heterogeneous computing device and the expected time-consuming value of the fault recovery includes: Obtaining a failure probability parameter of each of the heterogeneous computing devices; Sorting the fault probability parameters from large to small to obtain sorted fault probability parameters; Sorting the storage devices corresponding to the expected value of the fault recovery time to obtain the sorted expected value of the fault recovery time; The first checkpoint deployment relationship is determined by performing deployment processing on storage devices corresponding to the sorted failure probability parameters and the sorted expected values ​​of the failure recovery time.

7. The checkpoint deployment method for a heterogeneous computing system according to claim 6, characterized in that: Determining the first checkpoint deployment relationship by performing deployment processing on storage devices corresponding to the sorted failure probability parameters and the sorted expected values ​​of the failure recovery time includes: Obtain the checkpoint memory of the heterogeneous computing power device corresponding to the sorted failure probability parameters; Selecting the first fault probability parameter from the sorted fault probability parameters as the current fault probability parameter; The storage device corresponding to the last expected value of the fault recovery time after sorting is used as the current storage device; Deploy the checkpoint memory of the heterogeneous computing device corresponding to the current failure probability parameter to the current storage device; The storage space of the storage device corresponding to the expected time of the last failure recovery after the sorting after the deployment is updated; If the storage space of the storage device corresponding to the last expected value of the time required for fault recovery after sorting is greater than or equal to the checkpoint memory of the heterogeneous computing device corresponding to the second failure probability parameter, the second failure probability parameter in the sorted failure probability parameters is used as the new current failure probability parameter, and the process returns to the step of using the storage device corresponding to the last expected value of the time required for fault recovery after sorting as the current storage device; If the storage space of the storage device corresponding to the last expected value of the time taken for fault recovery after the sorting is smaller than the checkpoint memory of the heterogeneous computing device corresponding to the second fault probability parameter, the second fault probability parameter in the sorted fault probability parameters is used as the new current fault probability parameter; the storage device corresponding to the second-to-last expected value of the time taken for fault recovery after the sorting is used as the new current storage device; and the process returns to the step of deploying the checkpoint memory of the heterogeneous computing device corresponding to the current fault probability parameter to the current storage device for deployment until the checkpoint memory of all heterogeneous computing devices is deployed; Create a subset of the deployed heterogeneous computing devices corresponding to each storage device; A mapping relationship is established between each of the subsets and each of the storage devices to determine the first checkpoint deployment relationship.

8. The checkpoint deployment method for a heterogeneous computing system according to claim 7, characterized in that: The step of optimizing and determining the second checkpoint deployment relationship according to the deployment positions of the heterogeneous computing devices in the first checkpoint deployment relationship includes: The expected time-consuming values ​​of failure recovery corresponding to all storage devices are summed to determine a first optimization target value; Get the initial temperature value and the temperature reduction step; The initial temperature value is used as the current temperature value, and the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship are reassembled to the deployment position of the target storage device to obtain a third checkpoint deployment relationship; The new expected time-consuming values ​​of failure recovery corresponding to all storage devices in the third checkpoint deployment relationship are summed to determine a second optimization target value; When the first optimization target value is greater than the second optimization target value, the third checkpoint deployment relationship is used as a new first checkpoint deployment relationship; the current number of iterations is increased by 1, and the current temperature value is reduced by the temperature reduction step, and the step of recombining the heterogeneous computing devices in the subset corresponding to the storage devices of the first checkpoint deployment relationship to the deployment position of the target storage device to obtain the third checkpoint deployment relationship is returned to the step of recombining the heterogeneous computing devices in the subset corresponding to the storage devices of the first checkpoint deployment relationship to the deployment position of the target storage device to obtain the third checkpoint deployment relationship, until the reduced temperature value or the number of iterations meets the respective corresponding preset conditions, and the iteration is stopped to output the corresponding checkpoint deployment relationship as the second checkpoint deployment relationship; When the first optimization target value is less than or equal to the second optimization target value, obtain the acceptance probability and the random number, and determine the current checkpoint deployment relationship according to the random number and the acceptance probability; add 1 to the current number of iterations, reduce the current temperature value by the temperature reduction step, and return to the step of recombining the heterogeneous computing devices in the subset corresponding to the storage devices of the first checkpoint deployment relationship to the deployment position of the target storage device to obtain the third checkpoint deployment relationship, until the reduced temperature value or the number of iterations meets the corresponding preset conditions, stop the iteration, and output the corresponding checkpoint deployment relationship as the second checkpoint deployment relationship.

9. The checkpoint deployment method for a heterogeneous computing system according to claim 8, characterized in that: Recombining the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship to the deployment position of the target storage device to obtain a third checkpoint deployment relationship, including: The corresponding subsets of the heterogeneous computing devices corresponding to the checkpoint deployment in the same storage device are taken as the first set; wherein one storage device corresponds to one first set, and the first set corresponds to one heterogeneous computing device or multiple heterogeneous computing devices; Randomly selecting a target number of first sets in each first set; The heterogeneous computing devices in the first set of the target number are randomly replaced to determine the deployment relationship of the third checkpoint.

10. The checkpoint deployment method for a heterogeneous computing system according to claim 8, characterized in that: Recombining the heterogeneous computing devices in the subset corresponding to the storage devices in the first checkpoint deployment relationship to the deployment position of the target storage device to obtain a third checkpoint deployment relationship, including: The corresponding subsets of the heterogeneous computing devices corresponding to the checkpoint deployment in the same storage device are taken as the first set; wherein one storage device corresponds to one first set, and the first set corresponds to one heterogeneous computing device or multiple heterogeneous computing devices; Selecting a target first set including a plurality of heterogeneous computing devices from each first set; The heterogeneous computing devices of the first set where the adjacent target first set is located are randomly replaced to determine the third checkpoint deployment relationship.

11. The checkpoint deployment method for a heterogeneous computing system according to claim 1, characterized in that: After determining the deployment relationship of the second checkpoint, when a new training task arrives, it also includes: Get the memory storage path of the new training task; If the memory storage path is occupied by the checkpoint memory, the checkpoint is moved to a storage device in the storage device that corresponds to the minimum latency and has a memory space greater than or equal to the checkpoint memory.

12. A heterogeneous computing system, characterized in that: Includes a host, multiple heterogeneous computing devices, and multiple storage devices; The host is connected to each heterogeneous computing device respectively; each heterogeneous computing device calls each storage device; The host is used to execute the steps of the checkpoint deployment method for a heterogeneous computing system as described in any one of claims 1 to 11 above to complete the checkpoint deployment operation.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the checkpoint deployment method for a heterogeneous computing system described in any one of claims 1 to 11 are implemented.

14. A checkpoint deployment device for a heterogeneous computing system, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the checkpoint deployment method for a heterogeneous computing system as described in any one of claims 1 to 11 when executing the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the checkpoint deployment method for a heterogeneous computing system according to any one of claims 1 to 11 are implemented.

Images