Heterogeneous computing system and fault processing method, device, equipment, medium and product

Abstract

The application discloses a kind of heterogeneous computing systems and fault processing method, device, equipment, medium, product, it is related to heterogeneous computing technical field.Therein, method includes when the source computing device of heterogeneous computing system occurs failure in executing inference request task process, obtains the inference task information of inference request task and the computing power resource information of heterogeneous computing system, and determines the task estimated duration required for continuing to execute remaining task using the computing device without failure, and select the target computing device with the lowest task estimated duration from it.Task recovery request is sent to heterogeneous computing system, and the inference request task is migrated to target computing device by heterogeneous computing system to continue to execute.The application can solve the problem that related technology cannot reasonably handle failure and make the inference request task after recovery complete as soon as possible, can make the execution efficiency of inference request task after recovery as high as possible.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of heterogeneous computing, in particular to a heterogeneous computing system and a fault processing method, device, equipment, medium and product. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, the scale of network models processing artificial intelligence related tasks is getting larger and larger. In order to accelerate the task execution efficiency of large-scale network models, related technologies based on PD (Prefill-Decode, prefill-decode) separation inference method split the model and deploy it to multiple computing devices of a heterogeneous computing system. However, once a computing device fails, the related technology cannot reasonably handle the fault problem, and cannot make the recovered inference request task complete as soon as possible. SUMMARY

[0003] The present application provides a heterogeneous computing system and a fault processing method, device, electronic equipment, nonvolatile storage medium and computer program product, which can reasonably handle the fault problem when the PD separation inference process fails, and make the recovered inference request task complete as soon as possible, efficiently complete the inference request task.

[0004] To solve the above technical problems, the present application provides the following technical solutions:

[0005] In one aspect, the present application provides a fault processing method, comprising:

[0006] When a source computing device of a heterogeneous computing system fails during execution of an inference request task, inference task information of the inference request task and computing power resource information of the heterogeneous computing system are obtained. The source computing device is a computing device corresponding to a prefill stage and a decoding stage of the inference request task execution. The inference task information includes resource requirement information and source device information, and the computing power resource information includes performance parameters and fault monitoring information of each computing device.

[0007] According to the inference task information and the computing power resource information, the task estimated duration required for the remaining task of the inference request task to continue to be executed is determined by using the computing device without fault in the heterogeneous computing system. The target computing device with the lowest task estimated duration is selected, a task recovery request carrying at least the target computing device identifier and the task migration indication is generated, and is sent to the heterogeneous computing system to migrate the inference request task to the target computing device for continuous execution.

[0008] In another aspect, the present application provides a fault processing device, comprising:

[0009] The data acquisition module is configured to acquire inference task information of the inference request task and computing power resource information of the heterogeneous computing system when a source computing device of the heterogeneous computing system fails during execution of the inference request task, wherein the inference task information comprises resource requirement information and source device information, and the computing power resource information comprises performance parameters and fault monitoring information of each computing device.

[0010] The migration data determination module is configured to determine, according to the inference task information and the computing power resource information, a task estimation duration required for a computing device without failure in the heterogeneous computing system to continue to execute a remaining task of the inference request task.

[0011] The fault recovery module is configured to select a target computing device with the lowest task estimation duration, generate a task recovery request carrying at least an identifier of the target computing device and a task migration indication, and send the task recovery request to the heterogeneous computing system, so as to migrate the inference request task to the target computing device for continuous execution.

[0012] The application further provides an electronic device comprising a memory and a processor, wherein the processor is configured to implement the steps of any of the above fault processing methods when executing a computer program stored in the memory.

[0013] The application further provides a non-volatile storage medium, wherein the non-volatile storage medium stores a computer program, and the computer program is configured to implement the steps of any of the above fault processing methods when executed by a processor.

[0014] The application further provides a computer program product comprising a computer program / instruction, and the computer program / instruction is configured to implement the steps of any of the above fault processing methods when executed by a processor.

[0015] Finally, the application further provides a heterogeneous computing system comprising a plurality of computing devices and a processor, wherein each computing device is connected to the processor, and the processor is configured to implement the steps of any of the above fault processing methods when executing a computer program stored in a memory.

[0016] The technical scheme provided by the application has the advantages that once a computing device where an inference request task based on PD separation is located fails, the inference request task itself corresponding task related information, the resource and performance parameters of the heterogeneous computing system itself and the instantaneous information when the inference request task fails are integrated, the time length required for migrating the inference request task to other computing devices without failure to continue execution is determined, and the scheme with the lowest time consumption for continuing to execute the task is selected as the optimal recovery scheme, so that the optimal computing device for continuing to execute the task can be quickly locked, the execution efficiency of the recovered inference request task is as high as possible, the inference request task is executed in the shortest time after recovery, the fault recovery efficiency of the heterogeneous computing system is significantly improved, the final task processing result of the inference request task is obtained faster, the inference request task is efficiently completed, and the user experience under the condition of failure of the heterogeneous computing system is effectively improved.

[0017] In addition, the application also provides a corresponding implementation device, electronic equipment, nonvolatile storage medium and computer program product for the fault processing method, so that the method is more practical, and the device, electronic equipment, nonvolatile storage medium and computer program product have corresponding advantages. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical scheme of the application or related technology, the drawings needed in the embodiments or related technology description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creating any inventive labor.

[0019] Figure 1 The hardware composition framework diagram applicable to the fault processing method provided by the application.

[0020] Figure 2 The flowchart of the fault processing method provided by the application.

[0021] Figure 3 The framework diagram of the heterogeneous computing system provided by the application in an exemplary application scenario.

[0022] Figure 4 The schematic diagram of deploying PD separation type inference in the heterogeneous computing system provided by the application.

[0023] Figure 5 The structural framework diagram of the fault processing device provided by the application in an exemplary embodiment.

[0024] Figure 6 The structural diagram of the electronic equipment provided by the application in an exemplary embodiment.

[0025] Figure 7 A structural framework diagram of an exemplary embodiment of the heterogeneous computing system provided by the present application.

[0026] Figure 8 A framework diagram of an exemplary application scenario of the fault processing method provided by the present application. DETAILED DESCRIPTION

[0027] In order to enable persons skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. In the specification and the above drawings, the terms "first", "second", "third", "fourth" and the like are used to distinguish different objects, and are not intended to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment or illustration". Any embodiment described as "exemplary" herein is not necessarily interpreted as superior or better than other embodiments.

[0028] With the rapid development of artificial intelligence technology, more and more technical fields solve business problems in the field by training and deploying artificial intelligence network models, such as large language models (LLM). In order to meet the high-precision business processing needs of users, the network model for processing artificial intelligence related tasks is becoming larger and larger, and the demand for computing resources and storage resources of large-scale network models is also becoming larger and larger. In order to accelerate the inference speed of large-scale network models, distributed inference emerges as the times require. Distributed inference is to split a large-scale network model, and deploy the split model to multiple hardware devices with computing resources and storage resources, so as to realize distributed inference of large-scale models. Among them, PD (Prefill (pre-fill) Decode (decode)) separation inference is a distributed inference method that splits and deploys the pre-filling stage (which can be referred to as P stage) and the decoding stage (which can be referred to as D stage) in the inference process to different computing devices, such as deploying the pre-filling stage and the decoding stage to different graphic processors or instances, respectively optimizing scheduling and parallel strategy, so as to eliminate the competition between stages, significantly improve the throughput and response speed.

[0029] The heterogeneous computing system refers to integrating computing devices of multiple types of computing performance of the same physical host or multiple physical hosts in the same distributed computing environment, such as a multi-element heterogeneous computing system, each physical host can be defined as a computing node, and the computing devices with different computing performance cooperatively complete the distributed inference task of a large-scale network model. Inevitably, errors or faults will occur in the execution process of each computing device of the heterogeneous computing device, which may cause the running task to be interrupted and the user experience to rapidly decrease. In this case, the user's inference request task should be recovered as soon as possible through the checkpoint recovery mode and the computing is continued. However, due to the different computing performance of each computing device in the heterogeneous computing system, the interconnection network in the system is also heterogeneous, and each computing device has the possibility of re-fault, the related technology mostly focuses on the training task, does not consider the characteristics of the heterogeneous computing system and the PD separated large model inference task, and cannot reasonably handle the fault problem in the inference request task processing process, so that the recovered inference request task is completed as quickly as possible.

[0030] For example, one related technology serializes the model state obtained into a storable format to generate checkpoint data during the training process whenever checkpoint saving services are required at the current training round; then the checkpoint data is transmitted from the node internal device end memory to the host end memory. After the transmission is completed, the training task is continued, and asynchronous checkpoint saving is performed. When the training task fails, the task is recovered through the checkpoint data. This method is for training tasks and does not consider the characteristics of heterogeneous computing systems and large model inference tasks for PD separation. It cannot solve the problem of recovery when the inference request task fails during the execution of the heterogeneous computing system. Another related technology involves fault tolerance recovery of super-10000-core training and inference tasks. Training data is managed through a hierarchical storage structure, and when a computing node fails, global gradient backtracking is used for gradient recovery. During the recovery process, a dynamic time window is set based on the stored gradient snapshot, and the gradient recovery accuracy is optimized by adjusting the window length. When backtracking fails, a micro-checkpoint rollback is used to ensure stable recovery of the training process. Based on the causal dependency chain and gradient interpolation regression method, the key path is version-controlled and gradient replayed, and the non-key path is recovered by interpolation regression. Finally, the gradient recovery accuracy is improved through weighted correction. This method can reduce the gradient loss caused by computing node failure and improve the fault tolerance and computing efficiency of super-large-scale training tasks. However, this method does not consider the performance differences between different computing devices in the computing center, network communication, and other issues, so it has weak support for fault recovery in heterogeneous computing systems. Another related technology involves cloud-edge-end DNN (Deep Neural Network) collaborative recovery of inference interruption. If the resource service of an edge computing component and / or a cloud computing component is interrupted during the inference process of a DNN model in a collaborative inference system, a component that is currently not interrupted is selected as the current master component based on the principle of prioritizing edge computing components and cloud computing components. Based on the pre-stored DNN model data in the master component, a target inference recovery strategy is constructed for the current available resource amount of the master component to recover the inference of the DNN model in the collaborative inference system based on the strategy. This method can automatically recover the collaborative inference of the interrupted cloud-edge-end DNN and effectively improve the efficiency and reliability of inference recovery, thereby ensuring the reliability of inference. However, this method also does not consider the characteristics of different computing devices and PD separation. Although it solves the problem of inference task recovery, it cannot efficiently solve the problem of fault recovery in heterogeneous computing systems in the case of large model inference PD separation.

[0031] In view of this, based on the PD separation of the large model inference request device failure, according to the computing power, communication ability network information, failure rate information of different computing devices in the heterogeneous computing system, and the instantaneous information when the inference request fails, the optimal recovery scheme is calculated, so that the execution efficiency of the recovered inference request task is as high as possible, and the inference request task under the condition of failure of the heterogeneous computing system is efficiently completed, and the user experience is improved. Combined with the specific application environment architecture or specific hardware architecture on which the execution of the fault handling method depends, the specific application environment architecture or specific hardware architecture is described. The following describes the technical solutions of the present application in combination with Figure 1 Some possible application scenarios related to the technical solutions of the present application are exemplarily introduced, which can include the following contents:

[0032] The multi-element heterogeneous computing system includes multiple general servers and multiple artificial intelligence servers, each of which is equipped with multiple computing devices of different types. For example, the multi-element heterogeneous computing system can include 18 first-type servers each equipped with 4 graphic processing units, 12 second-type servers each equipped with 4 XPU (heterogeneous computing processor) chips, and 8 third-type servers each equipped with 4 tensor processing units. The different servers are connected through an exchange mechanism to build a non-100GB / s RoCEv2 (RDMA over Converged Ethernet Version 2, Remote Direct Memory Access over Converged Ethernet Version 2) lossless network, ensuring the stability of data transmission between the computing devices. The first-type servers can be used for high-parallel pre-padding stage calculation, for example, to adapt to input processing of large models. The second-type servers can be used for optimization decoding stage low-delay characteristics, for example, to meet the inference request service response demand. The third-type servers can be used as backup computing nodes, for example, to undertake fault migration tasks.

[0033] In addition to the management node 1, a high-performance server is selected from the plurality of heterogeneous computing systems as a fault handling node 2. The management node 1 is used to receive an inference request task sent by a user terminal 3, and is responsible for scheduling the processing of all inference request tasks, and feeds back the task processing result of each inference request task to the user. The management node 1 collects performance parameter data of the computing capability, communication capability and fault occurrence probability of each computing device of the heterogeneous computing system. For example, the computing capability of each computing device in the pre-filling stage and the decoding stage can be obtained, the probability value of each computing device executing a computing task request, then failing and executing a recalculation, and the bandwidth information between different computing devices. These performance parameters are stored as computing power resource information. When receiving an inference request task issued by a user, the inference task information corresponding to the inference request task is obtained, such as the computing amount of the inference request task in the pre-filling stage and the decoding stage, the amount of activation data transmitted between the pre-filling stage and the decoding stage, the related information of the source computing device allocated for the inference request task, such as the identification of the computing device, the computing capability, etc., which are stored as inference task information.

[0034] The management node 1 of the heterogeneous computing system allocates the pre-filling stage and the decoding stage of the inference request task to the source computing device, and monitors the executing inference request task in the process of executing the inference request task by using the source computing device. When detecting that the source computing device where the inference request task is located fails, the instantaneous information at the time of failure is obtained, such as the identification of the current failed source computing device, the stage where the current inference request task is located, the execution progress of the current inference request task in the current stage, and the checkpoint data size of the backup of the current inference request task, and these data are stored as fault monitoring information.

[0035] The fault handling node 2 detects the failure of the source computing device, obtains the inference task information, the performance parameters of each computing device and the fault monitoring information from the management node 1, calculates the remaining computing amount of the inference request task and the migration cost and data transmission cost of migrating it to an available computing device, determines a plurality of migration schemes according to the stage and the fault device, calculates the task estimated time corresponding to different migration schemes, selects the scheme with the lowest task estimated time as the optimal migration scheme, sends the identification of the target computing device determined by the optimal migration scheme and the task migration instruction to the management node 1 as a task recovery request, and the management node 1 migrates the inference request task to the target computing device for continuous execution after receiving the task recovery request sent by the fault handling node, and sends the final task processing result to the user terminal 3 after the inference request task is processed. When a fault occurs in the PD separation inference process, the fault problem can be reasonably handled, and the recovered inference request task can be completed as soon as possible, and the inference request task can be efficiently completed.

[0036] It should be noted that the above application scenarios are only shown for the purpose of facilitating the understanding of the ideas and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario. After introducing the technical solutions of the present application, various non-limiting embodiments of the present application will be described in detail in combination with the drawings and specific embodiments.

[0037] First, please refer to Figure 2 , according to the fault processing method provided by the present application, it can be implemented as a computer program product, which can be installed and run on one of the high-performance computing nodes or management nodes in the heterogeneous computing system or a special server outside the heterogeneous computing system, for implementing corresponding processing on the received user's inference request task, as shown in Figure 3 , which includes a plurality of heterogeneous computing devices, such as computer acceleration cards of different manufacturers and / or acceleration cards of different performance, which can communicate with each other within the same physical host, such as a server, or between different physical hosts. On the heterogeneous computing system, as shown in Figure 4 , before performing inference on the user's inference request task, the inference is first PD separated, and the pre-population stage and the decoding stage are executed in sequence to obtain the final task processing result. Among them, the pre-population stage processes the input of the inference request task and generates KV Cache (key value cache), which is relatively computationally intensive; the decoding stage generates the final result step by step according to the existing KV Cache, which is relatively sensitive to memory bandwidth. For example, as shown in Figure 4 , first, the pre-population stage calculation is performed in the first XPU, and then the intermediate data is transmitted to the second XPU for D stage calculation, and finally returned to the user. In Figure 4 , the common migration method when a fault occurs is also shown. During inference, the heterogeneous computing system will periodically save the checkpoint (checkpoint) of inference, which may include, for example, the intermediate results of model inference, the generated KV Cache and other intermediate data. Once it is found that the computing device where the inference request task is currently located has a fault, the checkpoint will be migrated to a new device to continue executing the inference task, but this method cannot ensure that the recovered inference request task can be completed as quickly as possible. The present embodiment can reasonably recover the inference request task in the heterogeneous computing system in the event of a fault and ensure that the inference request task is completed as quickly as possible, thereby improving the user's experience. In some embodiments of the method, the method includes the following steps:

[0038] S201: When the source computing device of the heterogeneous computing system fails during the execution of the inference request task, obtaining the inference task information of the inference request task and the computing power resource information of the heterogeneous computing system.

[0039] wherein the source computing device is one or two computing devices of the heterogeneous computing system, i.e., it is the computing device that performs the pre-population stage and the decoding stage of the inference request task before the failure, if the pre-population stage and the decoding stage use the same computing device, the source computing device is 1, if the pre-population stage and the decoding stage use two different computing devices, the source computing device includes two computing devices corresponding to the pre-population stage and the decoding stage, respectively. The failure is that the computing device corresponding to the pre-population stage or the computing device of the decoding stage in the source computing device fails, causing the inference request task to be interrupted.

[0040] wherein the inference request task can be any kind of user-issued and executed using a network model deployed in the heterogeneous computing system, an artificial intelligence related task, such as a natural language processing field inference task, a natural language processing inference task taking language data in the form of text / speech as input, understanding, generating, and analyzing human language through a natural language model deployed in the heterogeneous computing system, and being able to be applied to human-computer interaction, content processing, information extraction, etc. The inference request task can also be a computer vision field inference task, which takes “image / video data” as input, understands visual content through a visual model deployed in the heterogeneous computing system, realizes image recognition, target detection, scene analysis, etc., and can be applied to industry, security, medical treatment, etc. The inference request task can also be a speech and audio processing field inference task, which takes speech / audio signal as input, in addition to the speech-to-text task and text-to-speech task contained in the natural language task, it also includes speech emotion recognition, audio event detection, etc. “Audio semantic understanding” tasks, which can be applied to security, medical treatment, entertainment, etc. The inference request task can also be a recommendation system and personalized field inference task, which takes “user behavior data, item feature data” as input, predicts user preferences through an artificial intelligence model deployed in the heterogeneous computing system, realizes personalized recommendation, and is used in Internet platforms (such as e-commerce, short video, information). The inference request task can also be a decision and control field inference task, which takes “environmental data, state data” as input, outputs “decision instructions” or “control signals” through an artificial intelligence model deployed in the heterogeneous computing system, and directly drives device actions, which can be used in industrial automation, autonomous driving, robots, etc. The inference request task can also be a cross-modal inference task, which refers to an inference task whose input / output contains multiple modal data such as text, image, speech, and video, and realizes “semantic alignment” and “cooperative understanding” of different modal data through an artificial intelligence model deployed in the heterogeneous computing system.

[0041] In the embodiment, the inference task information is resource requirement information (such as pre-filling / decoding stage calculation amount, inter-stage data to be transmitted) and source device information (such as identification and computing power parameters of the source computing device) of the inference request task. The computing power resource information refers to fault monitoring information (such as fault device identification, target processing stage and corresponding task execution progress, checkpoint data amount, etc.) covering performance parameters (such as computing capability, fault probability, communication capability) of each computing device in the heterogeneous computing system and instantaneous information when reacting to faults. Among them, the performance parameters and the inference task information can be pre-stored to a specified location, and when a fault device is detected, a data acquisition operation is triggered to acquire the performance parameters and the inference task information.

[0042] S202: According to the inference task information and the computing power resource information, determine the task estimation duration required for the remaining tasks of the inference request task to be executed by using the computing devices without faults in the heterogeneous computing system.

[0043] Among them, the task estimation duration is the total time required for completing the remaining tasks of the inference request task by using the fault-free computing devices in the heterogeneous computing system, and the factors affecting the task estimation duration at least include the time required for the corresponding calculation amount of the remaining tasks, the time required for transferring intermediate data, and the time required for checkpoint transmission. Of course, if the migrated device also runs other tasks with high priority, it also needs to consider whether there is a task waiting for execution time, and if it does not consider the scenario of running other tasks or migrating tasks with the highest priority, it does not need to consider the waiting time of the computing device. The fault-free computing device can be a computing device without faults in the source computing device, a computing device with the highest performance and no running tasks in the heterogeneous computing system, a computing device with the highest performance but running tasks in the heterogeneous computing system, or a computing device reserved in advance as a backup in the heterogeneous computing system, which does not affect the implementation of the present application. As for the calculation of these times, it can be determined according to the performance parameters of the computing device and the corresponding calculation amount of the remaining tasks.

[0044] For example, when the source computing device executing large model PD separation inference in the heterogeneous computing system fails, first, the resource requirement information (such as the total calculation amount of the pre-filling stage is , the total calculation amount of the decoding stage is , and the pre-filling-decoding stage data to be transmitted is 2GB) and the source device information (such as the first source computing device is N100 (device identification), and the second source computing device is M370 (device identification)) of the inference request task are acquired; the computing power resource information is acquired, including the performance parameters of each computing device (such as the pre-filling stage calculation capability of N100 is , the fault probability is 0.02, the network bandwidth between each device is 10 GB / s, and the fault monitoring information (such as the fault device is the first source computing device, the fault occurs in the pre-filling stage, the task execution progress is 30%, and the checkpoint data volume is 1 GB). Based on the above information, all fault-free and non-executing computing devices are traversed, and the estimated time length of migrating the remaining task to each device is calculated respectively: for example, the estimated time length of migrating to N100 = checkpoint migration cost (1 GB / 10 GB / s = 0.1 s) + pre-filling remaining computing time ((1-30%) x / ÷(1-0.02)≈1.43s) + pre-filling-decoding data transmission cost (2 GB / 10 GB / s = 0.2 s) + decoding execution time (3 x / 3 x ÷(1-0.03)≈1.72s), the total time length is about 3.45 s; the estimated time length of migrating to M370 is calculated as 4.2 s.

[0045] S203: Select the target computing device with the lowest task estimated time length, generate a task recovery request carrying at least the target computing device identifier and the task migration indication, and send it to the heterogeneous computing system to migrate the inference request task to the target computing device for continuous execution.

[0046] Among them, the target computing device refers to the fault-free computing device with the lowest task estimated time length in the above steps, which is the receiver of the inference request task migration. The task recovery request is an instruction carrying the target computing device identifier and the task migration indication, which is used to trigger the heterogeneous computing system to perform the task migration operation. In the above example, N100 with the lowest task estimated time length is selected as the target computing device, and a task recovery request carrying N100 and the task migration indication is generated and sent to the management node of the heterogeneous computing system. The management node of the heterogeneous computing system migrates the inference request task of S201 to N100 for continuous execution.

[0047] In the technical scheme provided in the embodiment, once the computing device where the inference request task based on PD separation fails, the task related information corresponding to the inference request task itself, the resources and performance parameters of the heterogeneous computing system itself and the instantaneous information when the inference request task fails are integrated to determine the required duration for migrating the inference request task to other computing devices without failure to continue execution, and the scheme with the shortest duration for continuing execution of the task is selected as the optimal recovery scheme, so that the optimal computing device for continuing execution of the task can be quickly locked, the execution efficiency of the recovered inference request task is as high as possible, the recovered inference request task is executed in the shortest time, the fault recovery efficiency of the heterogeneous computing system is significantly improved, the final task processing result of the inference request task is obtained faster, the inference request task is efficiently completed, and the user experience under the condition of failure of the heterogeneous computing system is effectively improved.

[0048] In the above embodiment, how to perform step S202 is not limited, and an exemplary implementation manner for determining the time required for executing the remaining task by using the computing device without failure in the heterogeneous computing system is given in the embodiment, which can include the following contents:

[0049] According to the fault monitoring information and the resource requirement information, the remaining computing amount of the inference request task at the time of failure is determined, and the task recovery parameters of the candidate migration devices and the source computing device without failure for continuing execution of the inference request task are determined; the target processing stage of the inference request task at the time of failure is determined according to the fault monitoring information, and at least one alternative migration scheme is determined; in each alternative migration scheme, the task estimated duration required for migrating the inference request task to different computing devices of a matching type to continue execution is determined according to the remaining computing amount and the task recovery parameters.

[0050] In the embodiment, for the convenience of description, the computing devices of the heterogeneous computing system are divided into two categories, one is a source computing device, and the other is not a source computing device. Considering the execution of the task as fast as possible, the embodiment does not consider the computing device that is executing the task, and the category of the computing device is defined as a candidate migration device, that is, the computing device that is not a source computing device, is not in a task execution state, and is in a normal state in the heterogeneous computing system is taken as a candidate migration device. The remaining computing amount refers to the computing amount of the inference request task that is not completed at the time of failure. According to the stage at the time of failure, one scenario of the remaining computing amount is the pre-filling stage remaining computing amount + all computing amount of the decoding stage, and another scenario is the decoding stage remaining computing amount. The alternative migration scheme is a task migration strategy based on the target processing stage, and the device type of the device that continues to execute the inference request task is selected, which is the candidate migration device or the source computing device without failure, and is the device type for determining the task recovery (candidate migration device / source computing device without failure). The task recovery parameter is a parameter required for the estimated duration of the computing task, which can include the computing execution time of the computing device, the data migration cost, the inter-stage data transmission cost, and the like. The target processing stage refers to the execution stage of the inference request task at the time of failure, which is the pre-filling stage or the decoding stage.

[0051] For example, the first source computing device (device identifier N100) that is excluded from the heterogeneous computing system due to failure, the K920 (device identifier) processor that is executing other tasks, and the remaining M370 (device identifier) and R910 (device identifier) as candidate migration devices. According to the failure monitoring information, the failure occurs in the pre-filling stage (target processing stage), the total computing amount of the pre-filling stage is , and the execution progress is 30%, so the pre-filling remaining computing amount is = ; the decoding stage has not started execution, and the remaining computing amount is . The computing task recovery parameter: the pre-filling stage computing capability of M370 is , the failure probability is 0.03, the pre-filling execution time is ; the decoding stage computing capability is , and the decoding execution time is . The pre-filling computing capability of R910 is , the failure probability is 0.01, the pre-filling execution time is ; the decoding computing capability is , and the decoding execution time is The network bandwidth from the first source computing device to the M370 is 8GB / s, the checkpoint data volume is 1GB, and the migration cost is 1 GB / 8 GB / s = 0.125s; the bandwidth from the first source computing device to the R910 is 10GB / s, and the migration cost is 1 GB / 10 GB / s = 0.1s; the bandwidth between the M370 and the second source computing device (K920 processor) is 6GB / s, and the pre-filling-decoding data transmission cost is 2 GB / 6 GB / s ≈ 0.33s; the bandwidth between the R910 and the K920 processor is 9GB / s, and the transmission cost is 2 GB / 9 GB / s ≈ 0.22s. Because the fault occurs during the pre-filling stage, two alternative migration schemes are proposed: Scheme 1: Migrate to the candidate device to complete the pre-filling, and then have the second source device perform decoding; Scheme 2: Migrate to the second source device to complete the remaining pre-filling + decoding. The estimated task duration for each scheme is calculated as follows: Scheme 1 (M370): 0.125s + 2.42s + 0.33s + ( ÷(1-0.02))≈0.125+2.42+0.33+3.40≈6.275s; Scheme 1 (R910): 0.1s+1.77s+0.22s+3.40s≈5.49s; Scheme 2 (K920 processor): 0.15s (migration cost from the first source computing device to the K920 processor) + ( / + )÷(1-0.02)≈0.15+(3.5+3.33)÷0.98≈0.15+7.0≈7.15s.

[0052] As can be seen from the above, this embodiment can significantly improve the accuracy of task estimation by screening candidate devices in a hierarchical manner, accurately calculating the remaining computing power and task recovery parameters, and formulating differentiated alternative plans in combination with the fault stage, thus providing reliable data support for selecting the optimal migration strategy.

[0053] Based on the above embodiments, the present invention also provides a process for determining the remaining computational load, which may include the following:

[0054] Based on the fault monitoring information, determine the target processing stage at which the inference request task fails, and determine the total computational load of the pre-filling stage and the decoding stage of the inference request task based on the resource requirement information. If the target processing stage is the pre-filling stage, determine the remaining computational load of the pre-filling stage based on the total computational load of the pre-filling stage and the task execution progress, and determine the remaining computational load of the decoding stage based on the total computational load of the decoding stage of the inference request task. If the target processing stage is the decoding stage, there is no remaining computational load in the pre-filling stage of the inference request task, and determine the remaining computational load of the decoding stage based on the total computational load of the decoding stage and the task execution progress.

[0055] The pre-filling stage is the first stage of the inference model PD separated inference, and is used for processing input, generating KVCache, and computing intensive stage. The decoding stage is the second stage of the inference model PD separated inference, and is used for generating output based on the KV Cache, and is a memory bandwidth sensitive stage. The task execution progress refers to the proportion of the amount of computation completed in the current stage to the total amount of computation in the stage when the fault occurs, and the value range can be 0-1, for example.

[0056] As a convenient implementation manner, the P stage remaining computation amount calculation relationship and the D stage remaining computation amount calculation relationship can be stored in advance. If it is determined that the pre-filling stage is entered, the P stage remaining computation amount calculation relationship can be called The remaining computation amount of the pre-filling stage is determined, and the total computation amount of the stage is (D complete). If it is determined that the decoding stage is entered, the D stage remaining computation amount calculation relationship can be called The remaining computation amount of the decoding stage is determined, and the total computation amount of the stage is (P complete). Wherein, is the remaining computation amount of the pre-filling stage, is the remaining computation amount of the decoding stage, represents the task execution progress, represents the total computation amount of the P stage, represents the total computation amount of the D stage. For example, the inference request task fault occurs in the pre-filling stage, the total computation amount of the pre-filling stage is tested to be , the total computation amount of the decoding stage is estimated based on historical data to be , the task execution progress is 40%, and the remaining computation amount is . Because the fault occurs in the pre-filling stage, the decoding stage is not started, and the remaining computation amount is equal to the total computation amount, that is, . If the fault occurs in the decoding stage, the task execution progress is 20%, the pre-filling stage has no remaining computation amount, and the remaining computation amount of the decoding stage is .

[0057] As can be seen from the above, the embodiment calculates the remaining computation amount for the two stages of the PD separated inference respectively, can accurately match the task state when the fault occurs, avoids the computation amount statistical deviation, and provides accurate basic data for subsequent task duration estimation calculation.

[0058] Based on the above embodiment, the task recovery parameter at least includes execution time, and the application also provides a determination process of the execution time, which can include the following contents:

[0059] According to the computing resource information, the fault probability value of each candidate migration device and the source computing device without fault and the computing capacity of each candidate migration device and the source computing device without fault in the pre-filling stage and the decoding stage are determined, the first execution time of the pre-filling stage of each candidate migration device and the source computing device without fault for continuing to execute the inference request task is determined according to the first residual computing amount, the second execution time of the decoding stage of each candidate migration device and the source computing device without fault for continuing to execute the inference request task is determined according to the second residual computing amount, and the execution time of each candidate migration device and the source computing device without fault is determined according to the first execution time and the second execution time of each candidate migration device and the source computing device without fault for continuing to execute the inference request task.

[0060] Wherein, the residual computing amount includes the first residual computing amount of the pre-filling stage and the second residual computing amount of the decoding stage, that is, the first residual computing amount is the computing amount (FLOPs) not completed in the pre-filling stage, and the second residual computing amount is the computing amount (FLOPs) not completed in the decoding stage. The first execution time is the time required for the candidate migration device / the source computing device without fault to complete the residual computing amount in the pre-filling stage, which needs to consider the device fault probability. The second execution time is the time required for the candidate migration device / the source computing device without fault to complete the residual computing amount in the decoding stage, which needs to consider the device fault probability. The fault probability value refers to the probability of the computing device failing after executing a single task and needing to recalculate, which can be obtained by weighting the general fault probability and the historical fault probability.

[0061] As a convenient implementation manner, the P-stage execution time calculation relationship and the D-stage execution time calculation relationship can be pre-stored. For any one computing device i in the candidate migration device / the source computing device without fault, the P-stage execution time calculation relationship can be called to calculate the P-stage execution time of the computing device i The P-stage execution time of the computing device i The D-stage execution time of the computing device i can be calculated by calling the D-stage execution time calculation relationship The D-stage execution time of the computing device i Correspondingly, the total time of the P stage and the D stage of the computing device i can be calculated by calling the full-stage execution relationship The full-stage execution relationship can be expressed as Wherein, represents the computing capacity of the computing device i in the P stage, represents the fault probability value of the computing device i, represents the computing capacity of the computing device i in the D stage.

[0062] From the above, the embodiment can reflect the re-computation risk in the task execution process, make the execution time more suitable for the actual scene, avoid the task interruption after migration, improve the accuracy of the task estimated duration, and make the task execution efficiency after recovery as high as possible.

[0063] Based on the above embodiment, the task recovery parameter can also include the time required for migrating data during the task migration process. The application can use the migration cost and the data transmission cost as the task recovery parameter. The embodiment also provides a determination process of the migration cost and the data transmission cost, which can include the following contents:

[0064] According to the performance parameters of the computing devices, the network bandwidth between the failed device in the source computing device and each candidate migration device and the source computing device without failure is determined. According to the failure monitoring information, the data volume of the checkpoint backup data is determined. According to the data volume and the network bandwidth between the failed source computing device and each candidate migration device and the source computing device without failure, the migration cost of migrating the inference request task from the source computing device to each candidate migration device and the source computing device without failure is determined. According to the resource requirement information, the data volume to be transmitted in the conversion process of the inference request task in the pre-filling stage and the decoding stage is determined. According to the performance parameters of the computing devices, the network bandwidth between any two computing devices in each candidate migration device and the source computing device without failure is determined. According to the data volume to be transmitted, the data transmission cost between any two computing devices in each candidate migration device and the source computing device without failure is determined.

[0065] Among them, the pre-filling stage and the decoding stage of the inference request task correspond to different computing devices. The source computing device includes at least one failed device, that is, the source computing device includes a first source computing device and a second source computing device. The failed device can be the first source computing device and / or the second source computing device. The network bandwidth refers to the data transmission rate (unit: GB / s) between two computing devices. The checkpoint backup data is the intermediate data (such as KV Cache, model inference intermediate result) saved periodically during the execution of the inference task, which is used to recover the task when failure occurs. The data volume is in GB. The migration cost refers to the time required for migrating the checkpoint backup data from the failed source computing device to the candidate migration device or the source computing device without failure. It is equal to the checkpoint data volume / network bandwidth between the two computing devices (i.e., the source end and the destination end). The data volume to be transmitted is the activation data volume (unit: GB) that needs to be transmitted to the decoding stage device after the pre-filling stage is completed. The data transmission cost is the time required for transmitting the data to be transmitted between the pre-filling stage and the decoding stage in any two computing devices in each candidate migration device and the source computing device without failure. It is equal to the data volume to be transmitted / network bandwidth between the two devices.

[0066] Of course, if the source computing device is 1 or both source computing devices are faulty, the computing device corresponding to the failure stage is taken as the source computing device, the network bandwidth between the source computing device and each candidate migration device is determined according to the performance parameters of each computing device, the data volume of the checkpoint backup data is determined according to the failure monitoring information, and the migration cost of migrating the inference request task from the source computing device to each candidate migration device is determined according to the data volume and the network bandwidth between the source computing device and each candidate migration device; according to the resource requirement information, the data volume to be transmitted in the conversion process of the inference request task in the pre-padding stage and the decoding stage is determined, the network bandwidth between any two computing devices in each candidate migration device is determined according to the performance parameters of each computing device, and the data transmission cost between any two computing devices in each candidate migration device is determined according to the data volume to be transmitted.

[0067] As a convenient implementation manner, the migration cost calculation relationship and the data transmission cost calculation relationship can be stored in advance, each candidate migration device and the source computing device without failure can be defined as a computing device i, the migration cost calculation relationship is called The migration cost of the faulty source computing device to the computing device i is calculated In the relationship, src represents the source computing device without failure (i.e., the source end), tgt represents the computing device i (i.e., the destination end), represents the checkpoint backup data volume, represents the network bandwidth from the network bandwidth table, the migration cost of the faulty source computing device to the computing device i is calculated, and the data transmission cost calculation relationship is called The P-D conversion stage intermediate data transmission cost of the computing device i to any other available device j in each candidate migration device and the source computing device without failure is determined represents the data volume to be transmitted, represents the network bandwidth from the network bandwidth table, the migration cost of the faulty source computing device to the computing device i is calculated, and the data transmission cost calculation relationship is called represents the computing device i as a new P stage computing device, represents the computing device j as a new D stage computing device.

[0068] As can be seen from the above, the embodiment determines the time required for the computing device to perform the remaining task from two aspects of migration cost and transmission cost based on the actual network bandwidth, can accurately quantify the time loss of data migration and transmission, and makes the task estimation time more suitable for the actual running state of the heterogeneous computing system.

[0069] ​Based on the above embodiment, the embodiment also gives the implementation manner of how to determine the target processing stage of the inference request task when the fault occurs according to the fault monitoring information, and the implementation manner of determining the alternative migration scheme can include the following contents:

[0070] According to the fault monitoring information, if the target processing stage of the inference request task when the fault occurs is the pre-filling stage, the first alternative migration scheme is to migrate the checkpoint to one of the candidate migration devices to continue the pre-filling stage of the inference request task, and after the execution of the inference request task is completed, the decoding stage of the inference request task is executed by using the second source computing device; according to the fault monitoring information, if the target processing stage of the inference request task when the fault occurs is the pre-filling stage, the second alternative migration scheme is to migrate the checkpoint to the second source computing device, and to execute the remaining tasks of the pre-filling stage and the decoding stage of the inference request task by using the second source computing device; according to the fault monitoring information, if the target processing stage of the inference request task when the fault occurs is the decoding stage, the third alternative migration scheme is to migrate the checkpoint to one of the candidate migration devices or the first source computing device to continue the decoding stage of the inference request task.

[0071] In the embodiment, the source computing device includes a first source computing device corresponding to the pre-filling stage of the inference request task and a second source computing device corresponding to the decoding stage, the first source computing device and the second source computing device are different computing devices, the first source computing device is a computing device allocated for pre-filling stage execution. The second source computing device is a computing device allocated for decoding stage execution. The first alternative migration scheme is to migrate the checkpoint to the candidate device to complete the pre-filling when the fault occurs in the pre-filling stage, and then execute the decoding by the second source device. The second alternative migration scheme is to migrate the checkpoint to the second source device to complete the remaining pre-filling + full decoding when the fault occurs in the pre-filling stage. The third alternative migration scheme is to migrate the checkpoint to the candidate device to complete the remaining decoding when the fault occurs in the decoding stage.

[0072] As can be seen from the above, the embodiment formulates differentiated migration schemes for different fault stages, which can adapt to the task execution characteristics of the pre-filling and decoding stages, maximizes the use of fault-free device resources within the system, and improves the task recovery efficiency in different fault scenarios.

[0073] Based on the above embodiment, by integrating the time cost of each link of the first alternative migration scheme, the estimated task duration under the first alternative migration scheme can be accurately calculated, which provides accurate data for comparing the efficiency of different schemes. The determination process of the time required for the computing device under the first alternative migration scheme to execute the remaining tasks can include:

[0074] The migration cost of migrating the inference request task from the first source computing device to each candidate migration device is obtained; the first execution time corresponding to the remaining task of the pre-filling stage of each candidate migration device executing the inference request task is obtained, and the decoding execution time of the decoding stage of the second source computing device executing the inference request task is calculated; the data transmission cost between each candidate migration device and the second source computing device is obtained; the migration cost, the first execution time, the decoding execution time and the data transmission cost corresponding to each candidate migration device are respectively counted to obtain the task estimation time corresponding to each candidate migration device.

[0075] Wherein, the target processing stage of the inference request task failure is the pre-filling stage, and the first source computing device fails accordingly. The decoding execution time is the time required for the second source computing device to complete all the calculation amount of the decoding stage, and the device failure probability needs to be considered. The task estimation time corresponding to each candidate migration device is the sum of the corresponding migration cost, the first execution time, the decoding execution time and the data transmission cost. As a convenient implementation manner, the first expected time consumption calculation relationship can be pre-stored. For each candidate migration device adopting the first alternative migration scheme, the first expected time consumption calculation relationship can be called to calculate the task estimation time corresponding to each candidate migration device. The first expected time consumption calculation relationship can be expressed as: Wherein, represents the task estimation time corresponding to each candidate migration device of the first alternative migration scheme, is the id number of the failed device, which corresponds to the first source computing device, represents the id number of the new computing device of the P stage, is the id number of the second source computing device , which corresponds to the second source computing device, , , A represents the set of computing devices in the heterogeneous computing system except the failed device and the device executing the task, is the operator symbol in the computer representing not equal to, that is, it represents is each candidate migration device, and the entire time consumption includes the migration cost , the new computing device of the P stage continues to execute the remaining execution time corresponding to the P stage , the transmission network overhead (i.e. intermediate data) from the P stage to the D stage , the expected execution time of the second source computing device executing the D stage . When all the candidate migration devices are calculated, the minimum is selected from , and the device is recorded as the best migration device in the first alternative migration scheme i.e. if the first alternative migration scheme is adopted, the checkpoint is migrated to Continuing the P phase, after the P phase is completed, transmission to the second source computing device is performed to execute the D phase.

[0076] Based on the above embodiment, the second alternative scheme is explicitly determined, and the time length calculation logic is established to compare with the first alternative scheme, which can quickly determine the optimal device combination / single device to ensure the shortest task recovery time length. The determination process of the time length required for the computing device to execute the remaining task under the second alternative migration scheme can include:

[0077] obtaining a migration cost of migrating the inference request task from the first source computing device to the second source computing device; obtaining a first execution time corresponding to the remaining task of the pre-filling phase of the inference request task executed by the second source computing device, and calculating a decoding execution time of the decoding phase of the inference request task executed by the second source computing device; determining a task estimation time length corresponding to the second alternative migration scheme according to the migration cost, the first execution time and the decoding execution time. Correspondingly, if the task estimation time length corresponding to the second alternative migration scheme is less than the task estimation time length corresponding to each candidate migration device, the second source computing device is used as the target computing device for executing the inference request task; if the task estimation time length corresponding to the target candidate migration device is the smallest among the task estimation time lengths corresponding to all candidate migration devices and the task estimation time length corresponding to the second alternative migration scheme, the target candidate migration device is used as the target computing device for executing the pre-filling phase of the inference request task, and the second source computing device is used as the target computing device for executing the decoding phase of the inference request task.

[0078] Wherein, the target processing phase of the inference request task when it fails is the pre-filling phase, and correspondingly, the first source computing device fails, and the second source computing device completes all the remaining tasks, and the task estimation time length thereof is the sum of the corresponding migration cost, the first execution time and the decoding execution time. As a convenient implementation manner, the second expected time consumption calculation relationship can be pre-stored, and for the second alternative migration scheme, the second source computing device corresponding task estimation time length can be calculated by calling the second expected time consumption calculation relationship, and the time consumption of the second source computing device directly performing P and D phase calculation can be defined as i.e. for the second source computing device The first execution time of the remaining task of the pre-filling phase P and the decoding execution time of the decoding phase, and the second expected time consumption calculation relationship can be expressed as , represents the task estimation time length of the second source computing device under the second alternative migration scheme. I.e. if the second alternative migration scheme is adopted, the checkpoint is migrated to the second source computing device, and the remaining P phase and D phase are combined and executed on the second source computing device.

[0079] Based on the above embodiments, this embodiment formulates a special duration calculation and device selection method for decoding phase failure, which can quickly lock the optimal decoding task recovery device, and improve the recovery efficiency of decoding phase failure. The determination process of the time length required for the computing device under the third alternative migration scheme to execute the remaining tasks can include:

[0080] obtaining migration costs of migrating the inference request task from the second source computing device to each candidate migration device or the first source computing device respectively; obtaining second execution times of each candidate migration device and the first source computing device executing the remaining tasks of the decoding phase of the inference request task; respectively counting the migration costs and the second execution times corresponding to each candidate migration device and the first source computing device to obtain task estimated time lengths corresponding to each candidate migration device and the first source computing device respectively; accordingly, selecting the device with the minimum task estimated time length among all candidate migration devices and the first source computing device as the target computing device for executing the decoding phase of the inference request task.

[0081] Wherein, the target processing phase of the inference request task failure is the decoding phase, and accordingly, the second source computing device fails, and the first source computing device and each candidate migration device jointly serve as an alternative device for executing the remaining tasks of the decoding phase. The task estimated time length of each candidate migration device and the first source computing device under the third alternative migration scheme is the sum of the corresponding migration cost and the second execution time. As a convenient implementation manner, the third expected time consumption calculation relationship can be pre-stored. For the third alternative migration scheme, the third expected time consumption calculation relationship can be called to calculate the task estimated time length corresponding to each candidate migration device and the first source computing device, which represents the task estimated time length of each candidate migration device and the first source computing device under the third alternative migration scheme. The third expected time consumption calculation relationship can be represented as: . Wherein, In the third alternative migration scheme, the second source computing device, corresponds to one of the computing devices in the third alternative migration scheme, is the corresponding checkpoint migration time consumption cost. When the task estimated time lengths corresponding to all candidate migration devices and the first source computing device are calculated, the minimum is selected from them, and the device is recorded as the best migration device in the third alternative migration scheme That is, if the third alternative migration scheme is adopted, the checkpoint is migrated to and the remaining tasks of the D phase are continued.

[0082] In summary, when the inference request task fails, the target processing stage is the pre-filling stage. If the best migration device for the first alternative migration scheme... If the estimated task duration is less than or equal to the estimated task duration of the second source computing device under the second alternative migration scheme, then the optimal migration device is... The second source computing device serves as the target computing device for carrying the remaining tasks in the pre-filling stage, and the second source computing device serves as the target computing device for carrying the decoding stage; if the first alternative migration scheme's optimal migration device... If the estimated task duration is longer than the estimated task duration of the second source computing device under the second alternative migration scheme, then the second source computing device will be used as the target computing device to carry the remaining tasks of the pre-filling phase and the decoding phase; if the target processing phase when the inference request task fails is the decoding phase, then the best migration device under the third alternative migration scheme will be used. The target computing device that carries the remaining tasks of the decoding phase.

[0083] As can be seen from the above, in a multi-dimensional heterogeneous computing system composed of computing power from different manufacturers and with different performance, when the device containing the large model inference request based on PD separation fails, this embodiment calculates a reasonable recovery plan for the request, so that the computing system can complete the inference task as quickly as possible.

[0084] To quickly and accurately determine the target computing device, this invention also clarifies the data content contained in the inference task information. Accordingly, the process of obtaining the inference task information may include: obtaining the computational amount required by the inference request task in the pre-filling stage and the decoding stage respectively, and obtaining the amount of data to be transmitted during the conversion process of the inference request task in the pre-filling stage and the decoding stage, as resource requirement information; obtaining the first computing power information of the first source computing device allocated to the pre-filling stage of the inference request task, obtaining the second computing power information of the second source computing device allocated to the decoding stage of the inference request task, and using the first computing power information and the second computing power information as source device information.

[0085] The resource requirement information comprises the resource parameters needed to execute the inference request task, including the computational load of the pre-filling / decoding phase and the amount of data to be transmitted between phases. The source device information may include computing power information for both the first and second source computing devices. The first computing power information consists of the computing power parameters of the first source computing device, such as computing capacity, failure probability, and network bandwidth. The second computing power information consists of the computing power parameters of the second source computing device, and has the same dimensions as the first computing power information.

[0086] For example, the embodiment can also accurately calculate the amount of computation in the pre-filling and decoding stages by testing tasks, which can include the following: sending a reasoning test task of the same type of task as the reasoning request task to a target test computing device with the same performance parameters as the computing device currently executing the reasoning request task; obtaining the first test computation amount of the pre-filling stage and the second test computation amount of the decoding stage of the target test computing device by executing the reasoning test task corresponding to the reasoning model of the reasoning request task; determining the computation amount required for a single token length in the pre-filling stage and the decoding stage, respectively, according to the test token length, the first test computation amount, and the second test computation amount of the reasoning test task; and determining the computation amount required for the reasoning request task in the pre-filling stage and the decoding stage, respectively, according to the token length of the reasoning request task and the computation amount required for a single token length in the pre-filling stage and the decoding stage, respectively.

[0087] wherein the reasoning test task is a test reasoning task of the same type as the reasoning request task issued by the user, used to calculate the computation amount of a single token length. The target test computing device is a computing device with the same or similar performance parameters as the source computing device executing the reasoning request task, used to execute the reasoning test task. The first test computation amount is the total computation amount (FLOPs) of the target test computing device in the pre-filling stage of the reasoning test task. The second test computation amount is the total computation amount (FLOPs) of the target test computing device in the decoding stage of the reasoning test task. The test token length refers to the number of input tokens of the reasoning test task (e.g., 1000). The single token length computation amount refers to the computation amount required for a unit token in the pre-filling / decoding stage, which is equal to the test computation amount / test token length.

[0088] In this embodiment, for the inference model deployed by the heterogeneous computing system for performing the inference request task of S201, one computing device can be selected to receive the inference test task, which is an inference test request with token length A, input to the computing device, and the first target test device for performing the prefill stage and the second target test device for performing the decoding stage of the inference test task are selected in the heterogeneous computing system, the prefill process is performed on the first target test device, and the first test computation (FLOPs) of the process is collected by using a tool such as pytorch profiler during the execution process, and then the computation required for each token in the prefill stage is obtained by using the first test computation / A. After obtaining the above information, according to the number of tokens input by the inference request task, the computation of the inference request task in the P stage can be obtained by multiplying the computation required for each token in the prefill stage. In addition, the computation of the process can also be estimated by using various other methods such as mathematical formulas, which will not be described here. Similarly, the decoding stage is performed on the second target test device, and the second test computation (FLOPs) of the process is collected by using a tool such as pytorch profiler during the execution process, and the number of output tokens B is recorded, and then the computation required for each token in the decoding stage is obtained by using the second test computation / B. Subsequently, the average length of tokens output by the inference in a period of time is multiplied by the computation of each token to estimate the computation of the inference request task in the D stage. This information can be obtained by testing (executing a large model inference, monitoring and recording the network transmission data amount when the PD stage is converted), or directly estimated by the model structure, which will not be described here.

[0089] For example, a test task of the same type as the inference request task, such as a large language model text generation task, is constructed, and the input token length is set to 800. A target test device with the same model and configuration as the first source computing device (N100) is selected, and the test task is sent to the target test device. The first test computation of the prefill stage is monitored and recorded by pytorch profiler , and the second test computation of the decoding stage is 3.2x10¹¹ FLOPs. The computation of a single token is in the prefill stage and in the decoding stage. The token length of the inference request task is 1500, the prefill computation of the inference request task is , and the decoding computation of the inference request task is .

[0090] From the above, the embodiment can accurately deduce the stage calculation amount of the reasoning request task by calculating the calculation amount of a single word element through the same type of test task and the same performance test equipment, and greatly improve the accuracy of the calculation amount data.

[0091] To quickly and accurately determine the target computing device, the present application also specifies the data content contained in the performance parameters of each computing device in the computing power resource information. Accordingly, the process of obtaining the computing power resource information can include: obtaining each computing device of the heterogeneous computing system, performing the calculation capability of the pre-filling stage and the decoding stage of the reasoning test task belonging to the same task type as the reasoning request task; obtaining the fault probability value of each computing device of the heterogeneous computing system occurring failure and performing re-computation; obtaining the bandwidth information between each computing device of the heterogeneous computing system; taking the calculation capability of the pre-filling stage and the decoding stage, the fault probability value and the bandwidth information of each computing device of the heterogeneous computing system as the performance parameters of each computing device.

[0092] Among them, the reasoning test task is a standardized reasoning task for testing the pre-filling / decoding stage calculation capability of each computing device. The calculation capability refers to the calculation amount (FLOPs / s) that can be completed by the computing device per unit time, which is divided into pre-filling stage calculation capability and decoding stage calculation capability. The fault probability value is the probability of failure and re-computation of the computing device after performing a single reasoning task, which is obtained by weighting the general fault probability and the historical fault probability. The bandwidth information is the data transmission rate (GB / s) between any two computing devices in the heterogeneous computing system. The performance parameters are a parameter set containing the calculation capability, the fault probability value and the bandwidth information of the computing device. The calculation capability of each computing device of the heterogeneous computing system in the P stage and the D stage can be actually deployed and tested in each computing device, and the corresponding data can be obtained through the testing process. For the implementation method of obtaining the calculation amount of each stage of the reasoning request task through testing, the bandwidth information between different computing devices can be tested by writing a network test program, or the ideal value of each link can be used, for example, an adjacency matrix or adjacency table can be used to represent and store.

[0093] The embodiment can guarantee the authenticity and comprehensiveness of the computing power resource information by obtaining the calculation capability through standardized testing, calculating the fault probability by weighting, and obtaining the bandwidth information through actual testing, and provide a reliable data basis for the task recovery scheme.

[0094] For example, the implementation process of the weighted calculation of the failure probability can include: obtaining a general failure probability of each computing device of the heterogeneous computing system; determining a historical failure probability of each computing device according to historical failure data of each computing device of the heterogeneous computing system; and determining a failure probability value of each computing device according to the general failure probability and a corresponding weight coefficient of each computing device, the historical failure probability and a corresponding weight coefficient.

[0095] The general failure probability can be an average failure probability of the same type of computing device disclosed in the industry, which can be obtained from a manufacturer's manual or an industry report. In order to facilitate calculation, the obtained general failure probability can be initialized. The historical failure data can be determined according to the number of failures of the corresponding computing device in the heterogeneous computing system in a period of time (such as 3 months) and the total number of task executions, and the historical failure probability is the historical failure number / total number of task executions, reflecting the failure risk of the device in the actual running scene. The weight coefficient is the proportion of the general failure probability and the historical failure probability in the process of determining the failure probability, which can be adjusted according to the actual scene (such as a general probability weight of 0.5 and a historical failure probability weight of 0.5). Of course, the weight parameter can be adjusted according to the performance parameters and use time of the computing device. If it is a new device, the general weight parameter is larger, and if it is an older device, the historical weight parameter can be larger.

[0096] As can be seen from the above, the embodiment can take into account the industry commonality and actual running characteristics of the device by weighted calculation of the general failure probability and the historical failure probability, so that the failure probability value is more suitable for the actual scene of the heterogeneous computing system.

[0097] In order to quickly and accurately determine the target computing device, the present application also specifies the data content contained in the failure monitoring information in the computing power resource information. Correspondingly, the acquisition process of the failure monitoring information can include: when it is monitored that the computing device of the heterogeneous computing system fails in the process of executing the inference request task, obtaining the failure device identification information corresponding to the failure device; obtaining the target processing stage of the inference request task when the failure device fails, and determining the task execution progress of the inference request task in the target processing stage; obtaining the checkpoint backup data corresponding to the inference request task; and taking the failure device identification information, the target processing stage, the task execution progress and the data amount of the checkpoint backup data as the failure monitoring information.

[0098] The fault device identification information is information (such as a device model + a physical address, a device number) for uniquely identifying a fault computing device, and can be directly obtained. The target processing stage refers to an execution stage of an inference request task when a fault occurs, and is a pre-population stage or a decoding stage, and can be obtained by writing a monitoring program. The task execution progress is a proportion of a completed computation amount of a current stage to a total computation amount of the current stage when the fault occurs. The checkpoint backup data is intermediate data periodically saved during execution of the inference request task, and is used for resuming the task after the fault, and the data amount is in units of GB and can also be directly obtained. The fault monitoring information is a monitoring data set including the fault device identification, the target processing stage, the task execution progress, and the checkpoint data amount.

[0099] For example, the determination process of the execution progress of the current inference request task in the current stage can be: calculating the computation amount of each network neural layer of the inference model corresponding to the inference request task in the target processing stage, and determining the total computation amount; determining the target computation amount of the network neural layer already executed in the target processing stage, and determining the task execution progress according to the ratio of the target computation amount to the total computation amount.

[0100] The inference model is a network model for executing the inference request task, and is composed of multiple network neural layers. The network neural layer is a component unit of the inference model, and each layer corresponds to a certain computation amount (FLOPs). The total computation amount is the sum of the computation amounts of all network neural layers in the current processing stage (that is, pre-population or decoding), and the target computation amount refers to the sum of the computation amounts of the network neural layers already executed when the fault occurs. In this embodiment, the computation amount of each layer of the neural network can be calculated first, and then the execution progress can be obtained by dividing the computation amount of the neural layer already executed in the current stage by the total computation amount in the current stage, which can be realized through programming.

[0101] For example, taking the pre-population stage as an example, the inference model includes 20 network neural layers, and through model structure analysis and testing, the computation amount of each layer is calculated as , and the total computation amount is . Through the monitoring program, it is obtained that 8 layers of neural layers have been executed when the fault occurs, the target computation amount is , and the task execution progress can be .

[0102] As can be seen from the above, the execution progress is determined based on the neural layer computation amount of the inference model in this embodiment, which can solve the deviation caused by the time proportion estimation, accurately reflect the actual execution state of the task, and improve the accuracy of the remaining computation amount.

[0103] It should be noted that there is no strict execution order between the steps in the present application, as long as the logical order is met, the steps can be executed simultaneously, or executed in a certain preset order, Figure 2It is only a schematic way and does not mean that only such an execution sequence is possible.

[0104] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above-mentioned embodiments can be realized by means of software on a general hardware platform necessary, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment.

[0105] The application also provides a corresponding device for the fault processing method, which further makes the method more practical. Among them, the device can be explained from the perspective of functional modules and the perspective of hardware. The fault processing device provided by the application is introduced below. The device is used to realize the fault processing method provided by the application. In this embodiment, the fault processing device can include or be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to complete the fault processing method disclosed in embodiment one. The program module referred to in this embodiment refers to a series of computer program instruction segments that can complete a specific function. It is more suitable to describe the execution process of the fault processing device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module in this embodiment. The fault processing device described below can be referred to in conjunction with the fault processing method described above.

[0106] From the perspective of functional modules, please refer to Figure 5 , Figure 5 The structure diagram of the fault processing device provided in this embodiment can include:

[0107] The data acquisition module 501 is configured to acquire inference task information of the inference request task and computing power resource information of the heterogeneous computing system when a source computing device of the heterogeneous computing system fails during execution of the inference request task.

[0108] The migration data determination module 502 is configured to determine, according to the inference task information and the computing power resource information, a task estimation duration required for a computing device without failure in the heterogeneous computing system to continue executing a remaining task of the inference request task.

[0109] The fault recovery module 503 is configured to select a target computing device with the lowest task estimation duration, and generate a task recovery request carrying at least an identifier of the target computing device and a task migration indication, and send the task recovery request to the heterogeneous computing system, so as to migrate the inference request task to the target computing device for continuous execution.

[0110] In some embodiments of the present embodiment, the migration data determination module 502 can further be configured to: determine, as a candidate migration device, a computing device in the heterogeneous computing system that is not the source computing device, is not in the task execution state, and is in the normal state; determine, according to the fault monitoring information and the resource requirement information, a remaining computing amount of the inference request task at the time of the fault, and determine a task recovery parameter for each candidate migration device and the source computing device without the fault to continue executing the inference request task; determine, according to the fault monitoring information, a target processing stage at which the inference request task is at the time of the fault, and determine at least one alternative migration scheme; the alternative migration scheme is to select a device type to continue executing the inference request task; the device type is the candidate migration device or the source computing device without the fault; and in each alternative migration scheme, determine, according to the remaining computing amount and the task recovery parameter, a task estimation duration required for migrating the inference request task to a different computing device of the matching type to continue executing the inference request task to completion.

[0111] As an exemplary implementation of the above embodiment, the migration data determination module 502 can further be configured to: determine, according to the fault monitoring information, a target processing stage at which the inference request task is at the time of the fault, and determine, according to the resource requirement information, a total computing amount of each of a pre-filling stage and a decoding stage of the inference request task; if the target processing stage is the pre-filling stage, determine, according to the total computing amount of the pre-filling stage of the inference request task and a task execution progress, a remaining computing amount of the pre-filling stage, and determine, according to the total computing amount of the decoding stage of the inference request task, a remaining computing amount of the decoding stage; and if the target processing stage is the decoding stage, the pre-filling stage of the inference request task has no remaining computing amount, and the remaining computing amount of the decoding stage is determined according to the total computing amount of the decoding stage of the inference request task and the task execution progress.

[0112] As another exemplary implementation of the above embodiment, the migration data determination module 502 can further be configured to: the remaining computing amount includes a first remaining computing amount of the pre-filling stage and a second remaining computing amount of the decoding stage, determine, according to the computing power resource information, a fault probability value of each candidate migration device and the source computing device without the fault and a computing capability of each of the pre-filling stage and the decoding stage, determine, according to the first remaining computing amount, a first execution time corresponding to the pre-filling stage of each candidate migration device and the source computing device without the fault to continue executing the inference request task, and determine, according to the second remaining computing amount, a second execution time corresponding to the decoding stage of each candidate migration device and the source computing device without the fault to continue executing the inference request task; and determine, according to the first execution time and the second execution time of each candidate migration device and the source computing device without the fault to continue executing the inference request task, an execution time of each candidate migration device and the source computing device without the fault.

[0113] As another exemplary implementation of the above embodiment, the migration data determination module 502 can also be configured to: determine, according to the performance parameters of the computing devices, network bandwidths between the failed computing device in the source computing device and each candidate migration device and the source computing device without failure, determine, according to the failure monitoring information, a data amount of the checkpoint backup data, and determine, according to the data amount and the network bandwidths between the failed computing device in the source computing device and each candidate migration device and the source computing device without failure, migration costs of migrating the inference request task from the source computing device to each candidate migration device and the source computing device without failure, respectively; determine, according to the resource requirement information, a to-be-transmitted data amount in a conversion process of the inference request task in the pre-padding stage and the decoding stage, determine, according to the performance parameters of the computing devices, network bandwidths between any two computing devices among the candidate migration devices and the source computing device without failure, and determine, according to the to-be-transmitted data amount, data transmission costs between any two computing devices among the candidate migration devices and the source computing device without failure.

[0114] As another exemplary implementation of the above embodiment, the migration data determination module 502 can also be configured to: the source computing device includes a first source computing device corresponding to the pre-padding stage of the inference request task and a second source computing device corresponding to the decoding stage, if the target processing stage of the inference request task when the inference request task fails is determined to be the pre-padding stage according to the failure monitoring information, the first alternative migration scheme is to migrate the checkpoint to one of the candidate migration devices to continue to execute the pre-padding stage of the inference request task, and after the inference request task is executed, the decoding stage of the inference request task is executed by using the second source computing device; if the target processing stage of the inference request task when the inference request task fails is determined to be the pre-padding stage according to the failure monitoring information, the second alternative migration scheme is to migrate the checkpoint to the second source computing device, and the remaining task in the pre-padding stage of the inference request task and the decoding stage are executed by using the second source computing device; and if the target processing stage of the inference request task when the inference request task fails is determined to be the decoding stage according to the failure monitoring information, the third alternative migration scheme is to migrate the checkpoint to one of the candidate migration devices or the first source computing device to continue to execute the decoding stage of the inference request task.

[0115] As an exemplary implementation of the above embodiment, the migration data determination module 502 can also be configured to: when the alternative migration scheme is the first alternative migration scheme, obtain migration costs of migrating the inference request task from the first source computing device to each candidate migration device respectively; obtain a first execution time corresponding to a remaining task of the pre-padding stage of the inference request task executed by each candidate migration device, and calculate a decoding execution time of the decoding stage of the inference request task executed by the second source computing device; obtain a data transmission cost between each candidate migration device and the second source computing device; and respectively count the migration cost, the first execution time, the decoding execution time and the data transmission cost corresponding to each candidate migration device to obtain a task estimation time corresponding to each candidate migration device.

[0116] As another exemplary implementation of the above embodiment, the migration data determination module 502 can also be configured to: when the alternative migration scheme is the second alternative migration scheme, obtain a migration cost of migrating the inference request task from the first source computing device to the second source computing device; obtain a first execution time corresponding to a remaining task of the pre-padding stage of the inference request task executed by the second source computing device, and calculate a decoding execution time of the decoding stage of the inference request task executed by the second source computing device; determine a task estimation time corresponding to the second alternative migration scheme according to the migration cost, the first execution time and the decoding execution time; accordingly, if the task estimation time corresponding to the second alternative migration scheme is smaller than the task estimation time corresponding to each candidate migration device, the second source computing device is used as the target computing device for executing the inference request task; if the task estimation time corresponding to the target candidate migration device is the smallest among the task estimation times corresponding to all candidate migration devices and the task estimation time corresponding to the second alternative migration scheme, the target candidate migration device is used as the target computing device for executing the pre-padding stage of the inference request task, and the second source computing device is used as the target computing device for executing the decoding stage of the inference request task.

[0117] As another exemplary implementation of the above embodiment, the migration data determination module 502 can also be configured to: when the alternative migration scheme is the third alternative migration scheme, obtain migration costs of migrating the inference request task from the second source computing device to each candidate migration device or the first source computing device respectively; obtain a second execution time of a remaining task of the decoding stage of the inference request task executed by each candidate migration device and the first source computing device; respectively count the migration cost and the second execution time corresponding to each candidate migration device and the first source computing device to obtain a task estimation time corresponding to each of the candidate migration devices and the first source computing device; accordingly, select the device with the smallest task estimation time among all candidate migration devices and the first source computing device as the target computing device for executing the decoding stage of the inference request task.

[0118] Exemplarily, in some other embodiments of the present embodiment, the data acquisition module 501 can also be configured to: acquire the computation amount required by the inference request task in the pre-filling stage and the decoding stage respectively, and acquire the amount of data to be transmitted in the conversion process of the inference request task in the pre-filling stage and the decoding stage, as the resource requirement information; acquire the first computing power information of the first source computing device allocated corresponding to the pre-filling stage of the inference request task, acquire the second computing power information of the second source computing device allocated corresponding to the decoding stage of the inference request task, and take the first computing power information and the second computing power information as the source device information.

[0119] As an exemplary implementation of the above embodiment, the data acquisition module 501 can also be configured to: send an inference test task belonging to the same task type as the inference request task to a target test computing device with the same computing device performance parameter as the computing device currently executing the inference request task; acquire a first test computation amount of the target test computing device executing the pre-filling stage of the inference test task by the inference model corresponding to the inference request task, and a second test computation amount of the target test computing device executing the decoding stage of the inference test task; determine the computation amount required by a single token length in the pre-filling stage and the decoding stage respectively according to the test token length of the inference test task, the first test computation amount and the second test computation amount; and determine the computation amount required by the inference request task in the pre-filling stage and the decoding stage according to the token length of the inference request task and the computation amount required by a single token length in the pre-filling stage and the decoding stage respectively.

[0120] Exemplarily, in some other embodiments of the present embodiment, the data acquisition module 501 can also be configured to: acquire the computing capability of each computing device of the heterogeneous computing system executing the pre-filling stage and the decoding stage of the inference test task belonging to the same task type as the inference request task; acquire the failure probability value of each computing device of the heterogeneous computing system failing and executing re-computation; acquire the bandwidth information between each computing device of the heterogeneous computing system; and take the computing capability of each computing device of the heterogeneous computing system in the pre-filling stage and the decoding stage, the failure probability value and the bandwidth information as the performance parameter of each computing device.

[0121] As an exemplary implementation of the above embodiment, the data acquisition module 501 can also be configured to: acquire the general failure probability of each computing device of the heterogeneous computing system; determine the historical failure probability of each computing device according to the historical failure data of each computing device of the heterogeneous computing system; and determine the failure probability value of each computing device according to the general failure probability of each computing device and the corresponding weight coefficient, the historical failure probability of each computing device and the corresponding weight coefficient.

[0122] Exemplarily, in some other embodiments of the present embodiment, the data acquisition module 501 can also be configured to: when it is monitored that a computing device of the heterogeneous computing system fails during execution of the inference request task, acquire failure device identification information corresponding to the failed device; acquire a target processing stage at which the inference request task is located when the failed device fails, and determine a task execution progress of the inference request task at the target processing stage; acquire checkpoint backup data corresponding to the inference request task; and take the failure device identification information, the target processing stage, the task execution progress and a data amount of the checkpoint backup data as the failure monitoring information.

[0123] As an exemplary implementation of the above embodiment, the data acquisition module 501 can also be configured to: calculate a total amount of computation of each network neural layer of the inference model corresponding to the inference request task at the target processing stage, and determine the total amount of computation; determine a target amount of computation of the network neural layer that has been executed at the target processing stage, and determine the task execution progress according to a ratio of the target amount of computation to the total amount of computation.

[0124] The failure processing apparatus mentioned above is described from the perspective of functional modules, and further, the present embodiment also provides an electronic device, which is described from the perspective of hardware. Figure 6 The electronic device provided by the present embodiment has a structure as shown in the structural schematic diagram of an embodiment. The electronic device includes a memory 601 and a processor 602, the memory 601 stores a computer program, and the processor 602 is configured to run the computer program to execute the steps in any of the above failure processing method embodiments.

[0125] The present embodiment also provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above failure processing method embodiments when running.

[0126] In an exemplary embodiment, the computer readable storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0127] The present embodiment also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps in any of the above failure processing method embodiments.

[0128] The embodiment of the present application also provides another computer program product, comprising a nonvolatile computer readable storage medium, the nonvolatile computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps in any of the above fault processing method embodiments.

[0129] Finally, the present application also provides a heterogeneous computing system, please refer to Figure 7 , which can comprise a plurality of computing devices 701 and a processor 602, each of the computing devices 701 being connected to the processor 602; the processor 602 is configured to execute a computer program stored in a memory to implement the steps in any of the above fault processing method embodiments. Among them, the computing performance of the computing device 701 is different, and these computing devices 701 collectively constitute the heterogeneous computing resources in the heterogeneous computing system, and each computing device 701 can be inserted into different computing nodes (such as servers) or the same computing node, which does not affect the implementation of the present application. The processor 602 can be deployed on one of the high-performance computing nodes or management nodes in the heterogeneous computing system or a dedicated server outside the heterogeneous computing system, which does not affect the implementation of the present application.

[0130] In order to make the skilled in the art more clearly understand how to solve the problem of the fault of the computing device in the heterogeneous computing system, the present application also gives an exemplary implementation, which can include the following contents:

[0131] In the embodiment, the computer program for executing the fault processing method is packaged into four functional modules: an inference task information collection module, a heterogeneous computing system and real-time task information collection module, a task recovery scheme output module and a recovery scheme implementation module. Among them, the task recovery scheme output module has an input receiving layer, an execution layer and an output layer, the receiving layer receives the corresponding data sent by the inference task information collection module and the heterogeneous computing system and real-time task information collection module, and the recovery scheme implementation module is configured to receive the output data of the output layer. The inference task information collection module can be deployed on the computing node for receiving the inference request task issued by the user in the heterogeneous computing system, the heterogeneous computing system and real-time task information collection module can be deployed on the management node of the heterogeneous computing system, the task recovery scheme output module can be deployed on the high-performance computing node in the heterogeneous computing system, and the recovery scheme implementation module can also be deployed on the management node.

[0132] Among them, as Figure 8As shown, the inference task information collection module is configured to collect inference task information corresponding to the inference request task issued by the user and send the inference task information to the task recovery scheme output module. The heterogeneous computing system and the real-time task information collection module are configured to collect performance parameter information of the computing devices of the heterogeneous computing system and monitor the inference request task being executed. When a fault occurs, the fault monitoring information is collected, and the fault monitoring information and the performance parameter information of the computing devices are packaged and sent to the task recovery scheme output module. The recovery scheme implementation module is configured to perform actual issuance and deployment in the heterogeneous computing system. According to the target computing device identifier output by the task recovery scheme output module, the recovery scheme implementation module generates a task recovery request indicating that the inference request task is migrated to the target computing device. The management node acquires the task identifier corresponding to the inference request task, and migrates the inference request task to the target computing device through the task identifier and the task recovery request.

[0133] The input layer of the task recovery scheme output module receives a set of inference task information parameters and a set of computing power resource information parameters. The set of inference task information parameters includes current_stage (target processing stage (string, or , indicating that the fault occurs in the P stage or the D stage), (total computation amount (FLOPs) of the P stage of the inference request task), (total computation amount (FLOPs) of the D stage of the inference request task), and progress (task execution progress), which is a floating-point number and has a value range of: , indicating the completed proportion and the remaining computation amount at the time of the fault , where curr is P or D, checkpoint_size (i.e., checkpoint backup data amount), and intermediate_data_size (i.e., intermediate data amount in the P-D stage conversion). The set of computing power resource information parameters includes Devices (a list of computing devices), which contains the computing power of each computing device in the P and D stages, respectively, (FLOPs / s), fault probability , P_dev_id (device id of the first source computing device used in the original P stage), D_dev_id (device id corresponding to the second source computing device used in the original D stage), fault_dev_id (device id of the current fault device), and bandwidth_table (bandwidth table), which can be a dictionary with (src_id, tgt_id) pairs as keys, indicating the network bandwidth between the source end src and the destination end tgt.

[0134] The execution layer includes a preprocessing stage, a time-consuming estimation stage, and a device selection stage.

[0135] The preprocessing stage encapsulates the calculation formulas for the remaining computational load of stage P, the remaining computational load of stage D, the execution time of stage P, the execution time of stage D, the execution relationship of all stages, the migration cost calculation formula, and the data transmission cost calculation formula. It also determines the target processing stage at the time of the fault, i.e., the current_stage. ), identify the faulty device fault_dev_id, remove the faulty device and the device that is currently performing a task from Devices, and obtain the set of available devices A.

[0136] if Then, the remaining computational amount in the pre-filling stage is determined by calling the formula for calculating the remaining computational amount in stage P. (D complete). If ,but (P completed) The remaining computational load for the decoding stage is determined by calling the remaining computational load calculation formula for stage D. For each available device i in the available device set A, the execution time of each available device i in A during stages P, D, and the entire stage is calculated by calling the execution time calculation formulas for stages P, D, and the entire stage. Taking the faulty device as the source and each available device i in the available device set A as the destination, the migration cost from the faulty device (fault_dev) to each available device i is determined by calling the migration cost calculation formula. For any two available devices i and j in the available device set A, the intermediate data transmission cost during the PD conversion stage from device i to any other available device j is determined by calling the data transmission cost calculation formula.

[0137] The time estimation stage may include: pre-encapsulating a first expected time calculation formula, a second expected time calculation formula, and a third expected time calculation formula. If... Then, by calling the first expected time calculation formula, the estimated task duration for each available device i in the available device set A under the first alternative migration scheme, excluding the second source fault device, is calculated. Then, by calling the second expected time calculation formula, the estimated task duration for the second source fault device under the second alternative migration scheme is calculated. If... Then, the third expected time calculation formula is invoked to calculate the estimated task duration for each available device in the available device set A.

[0138] The equipment selection phase may include: if Then, select the best migration device with the lowest estimated task duration from the first alternative migration plan. .like If the estimated task duration is less than or equal to the estimated task duration of the second source computing device under the second alternative migration scheme, then output... The device identifier and task identifier corresponding to the second source computing device, wherein the task identifier is an indicator. The target computing device for carrying the remaining tasks in the pre-filling stage, and the second source computing device for carrying the target computing device in the decoding stage, are as follows: and .like The estimated task duration is greater than the estimated task duration of the second source computing device under the second alternative migration scheme. The second source computing device serves as the target computing device for carrying the remaining tasks in the pre-filling phase and the decoding phase. The output includes the device identifier and the corresponding task identifier of the second source computing device, as shown below. .if Then, select the best migration device with the lowest estimated task duration from the third alternative migration plan. The best migration device for using the third alternative migration option The target computing device that carries the remaining tasks in the decoding phase outputs... The corresponding device identifier and the corresponding task identifier are as follows: .

[0139] As shown above, in this embodiment, within a multi-dimensional heterogeneous computing system composed of computing power from different manufacturers and with varying performance levels, when the computing device hosting a large model inference request task based on PD separation fails, a reasonable task recovery plan is determined for that inference request task to maximize the speed at which the heterogeneous computing system completes the inference task. Furthermore, based on the performance parameters of the computing devices in the heterogeneous computing system, the network connectivity between different computing devices, the failure rate information of the computing devices, and the instantaneous fault monitoring information when the inference request task fails, the optimal task recovery plan is determined to maximize the execution efficiency of the recovered inference request task, effectively improving the processing efficiency of the inference request task and enabling users to obtain the processing results of the inference request task faster, thereby maximizing the user experience in the event of a failure.

[0140] The above describes in detail the heterogeneous computing system and the fault processing method, device, electronic equipment, nonvolatile storage medium, and computer program product provided by the present application. Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be understood by referring to each other. The units and algorithm steps of each example described in each disclosed embodiment are executed in the form of electronic hardware or computer software, which depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, and such implementation should not be considered beyond the scope of the present application. Without departing from the principles of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the scope of the present application.

Claims

1. A fault handling method, characterized in that, include: When the source computing device of the heterogeneous computing system malfunctions during the execution of the inference request task, the inference task information of the inference request task and the computing power resource information of the heterogeneous computing system are obtained. Based on the inference task information and the computing power resource information, determine the estimated time required to continue executing the remaining tasks of the inference request task using the fault-free computing devices in the heterogeneous computing system. Select the target computing device with the lowest estimated task duration, generate a task recovery request that carries at least the target computing device identifier and task migration instruction, and send it to the heterogeneous computing system to migrate the inference request task to the target computing device for continued execution; The source computing device is the computing device corresponding to the pre-filling and decoding phases of the inference request task, including a fault-free computing device; the inference task information includes resource requirement information and source device information, and the computing power resource information includes fault monitoring information and performance parameters of each computing device; computing devices in the heterogeneous computing system that are not the source computing device, are not in a task execution state, and are in a normal state are selected as candidate migration devices; based on the fault monitoring information and the resource requirement information, task recovery parameters for each candidate migration device and the fault-free source computing device to continue executing the inference request task are determined, and the task recovery parameter determination process includes: Based on the performance parameters of each computing device, determine the network bandwidth between the faulty device in the source computing device and each candidate migration device and the fault-free source computing device. Based on the fault monitoring information, determine the amount of checkpoint backup data. Based on the data volume and the network bandwidth between the source computing device and each candidate migration device and the fault-free source computing device, determine the migration cost of migrating the inference request task from the source computing device to each candidate migration device and the fault-free source computing device. Based on the resource requirement information, determine the amount of data to be transmitted by the inference request task during the pre-filling and decoding conversion processes. Based on the performance parameters of each computing device, determine the network bandwidth between any two computing devices among the candidate migration devices and the fault-free source computing devices. Based on the amount of data to be transmitted, determine the data transmission cost between any two computing devices among the candidate migration devices and the fault-free source computing devices.

2. The fault handling method according to claim 1, characterized in that, Determine the estimated time required to complete the remaining tasks of the inference request task using fault-free computing devices in the heterogeneous computing system, including: Based on the fault monitoring information and the resource requirement information, the remaining computational load of the inference request task in the event of a fault is determined; the source computing device includes a non-faulty computing device and a faulty computing device. Based on the fault monitoring information, determine the target processing stage at which the inference request task fails, and determine at least one alternative migration scheme; the alternative migration scheme determines the device type to continue executing the inference request task based on different target processing stages; the device type is a candidate migration device or a source computing device without faults; In each alternative migration scheme, based on the remaining computational load and the task recovery parameters, the estimated task duration required to migrate the inference request task to a different computing device of a matching type for continued execution is determined.

3. The fault handling method according to claim 2, characterized in that, Based on the fault monitoring information and the resource requirement information, the remaining computational load of the inference request task during a fault is determined, including: The target processing stage at which the inference request task fails is determined based on the fault monitoring information, and the total computational load of the pre-filling stage and the decoding stage of the inference request task is determined based on the resource requirement information. If the target processing stage is a pre-filling stage, then the remaining computational amount of the pre-filling stage is determined based on the total computational amount of the pre-filling stage of the inference request task and the task execution progress, and the remaining computational amount of the decoding stage is determined based on the total computational amount of the decoding stage of the inference request task. If the target processing stage is the decoding stage, then there is no remaining computation in the pre-filling stage of the inference request task. The remaining computation in the decoding stage is determined based on the total computation in the decoding stage of the inference request task and the task execution progress.

4. The fault handling method according to claim 2, characterized in that, The remaining computational load includes the first remaining computational load of the pre-filling phase and the second remaining computational load of the decoding phase. Task recovery parameters are used to determine whether each candidate migration device and the fault-free source computing device can continue executing the inference request task, including: Based on the computing power resource information, determine the failure probability value of each candidate migration device and the source computing device without failure, as well as their computing capabilities in the pre-filling stage and the decoding stage respectively. Based on the first remaining computing power, determine the first execution time corresponding to the pre-filling stage of the inference request task for each candidate migration device and the source computing device without failure. Based on the second remaining computing power, determine the second execution time corresponding to the decoding stage of the inference request task for each candidate migration device and the source computing device without failure. The execution time of each candidate migration device and the source computing device without fault is determined based on the first execution time and the second execution time of each candidate migration device and the source computing device without fault continuing to execute the inference request task.

5. The fault handling method according to claim 2, characterized in that, The pre-filling and decoding phases of the inference request task correspond to different computing devices.

6. The fault handling method according to claim 2, characterized in that, The source computing devices include a first source computing device corresponding to the pre-filling phase of the inference request task and a second source computing device corresponding to the decoding phase. Based on the fault monitoring information, the target processing phase at which the inference request task fails is determined, and at least one alternative migration scheme is determined, including: If the target processing stage at which the inference request task fails is determined to be the pre-filling stage based on the fault monitoring information, then the first alternative migration scheme is: to migrate the checkpoint to one of the candidate migration devices to continue executing the pre-filling stage of the inference request task, and after the inference request task is completed, to use the second source computing device to execute the decoding stage of the inference request task. If the target processing stage at which the inference request task fails is determined to be the pre-filling stage based on the fault monitoring information, then the second alternative migration scheme is: to migrate the checkpoint to the second source computing device and use the second source computing device to execute the remaining tasks of the pre-filling stage and the decoding stage of the inference request task. If the target processing stage of the inference request task is determined to be the decoding stage when the fault monitoring information is determined to be the faulty inference request task, then the third alternative migration scheme is: to migrate the checkpoint to one of the candidate migration devices or the first source computing device to continue to execute the decoding stage of the inference request task.

7. The fault handling method according to claim 6, characterized in that, The alternative migration plan is the first alternative migration plan, which determines the estimated time required to migrate the inference request task to a different computing device of a matching type for continued execution, including: Obtain the migration cost of migrating the inference request task from the first source computing device to each candidate migration device; Obtain the first execution time corresponding to the remaining tasks in the pre-filling phase of the inference request task executed by each candidate migration device, and calculate the decoding execution time of the decoding phase of the inference request task executed by the second source computing device. Obtain the data transfer cost between each candidate migration device and the second source computing device; The migration cost, first execution time, decoding execution time and data transmission cost for each candidate migration device are calculated separately to obtain the estimated task duration for each candidate migration device.

8. The fault handling method according to claim 6, characterized in that, The alternative migration plan is a second alternative migration plan, which determines the estimated time required to migrate the inference request task to a different computing device of a matching type for continued execution, including: Obtain the migration cost of migrating the inference request task from the first source computing device to the second source computing device; Obtain the first execution time corresponding to the remaining tasks in the pre-filling phase of the inference request task executed by the second source computing device, and calculate the decoding execution time of the decoding phase of the inference request task executed by the second source computing device. The estimated task duration corresponding to the second alternative migration scheme is determined based on the migration cost, the first execution time, and the decoding execution time. Accordingly, if the estimated task duration corresponding to the second alternative migration scheme is less than the estimated task duration corresponding to each candidate migration device, then the second source computing device will be used as the target computing device for executing the inference request task. If the estimated task duration corresponding to the target candidate migration device is the smallest among the estimated task durations corresponding to all candidate migration devices and the estimated task durations corresponding to the second alternative migration scheme, then the target candidate migration device is used as the target computing device for the pre-filling stage of the inference request task, and the second source computing device is used as the target computing device for the decoding stage of the inference request task.

9. The fault handling method according to claim 6, characterized in that, The alternative migration plan is the third alternative migration plan, which determines the estimated time required to migrate the inference request task to a different computing device of a matching type for continued execution, including: Obtain the migration cost of migrating the inference request task from the second source computing device to each candidate migration device or the first source computing device; Obtain the second execution time of the remaining tasks in the decoding phase of the inference request task for each candidate migration device and the first source computing device; The migration cost and second execution time corresponding to each candidate migration device and the first source computing device are calculated separately to obtain the estimated task duration for each candidate migration device and the first source computing device. Accordingly, the device with the smallest estimated task time among all candidate migration devices and the first source computing device is selected as the target computing device for executing the decoding phase of the inference request task.

10. The fault handling method according to any one of claims 1 to 9, characterized in that, Obtain the inference task information for the inference request task, including: The computational amount required by the inference request task in the pre-filling stage and the decoding stage is obtained, and the amount of data to be transmitted by the inference request task in the conversion process of the pre-filling stage and the decoding stage is obtained as resource requirement information. Obtain the first computing power information of the first source computing device allocated in the pre-filling stage of the inference request task, obtain the second computing power information of the second source computing device allocated in the decoding stage of the inference request task, and use the first computing power information and the second computing power information as source device information.

11. The fault handling method according to claim 10, characterized in that, The computational requirements of the inference request task in the pre-filling and decoding stages are obtained, including: The inference test task, which belongs to the same task type as the inference request task, is sent to the target test computing device with the same performance parameters as the computing device currently executing the inference request task. The target test computing device obtains the first test computation amount for the pre-filling stage of the inference test task and the second test computation amount for the decoding stage of the inference test task through the inference model corresponding to the inference request task. Based on the test lexical length of the inference test task, the computational cost of the first test, and the computational cost of the second test, determine the computational cost required for a single lexical length in the pre-filling stage and the decoding stage, respectively. Based on the computational requirements of the lexical length and single lexical length of the inference request task in the pre-filling and decoding stages, respectively, the computational requirements of the inference request task in the pre-filling and decoding stages are determined.

12. The fault handling method according to any one of claims 1 to 9, characterized in that, The process of acquiring computing resource information of the heterogeneous computing system includes: The computing capabilities of each computing device in the heterogeneous computing system to perform the pre-filling and decoding phases of an inference test task that belongs to the same task type as the inference request task are obtained. Obtain the failure probability value of each computing device in the heterogeneous computing system that fails and requires recalculation; Obtain bandwidth information between computing devices in the heterogeneous computing system; use the computing power, failure probability value, and bandwidth information of each computing device in the pre-filling and decoding stages as performance parameters of each computing device.

13. The fault handling method according to claim 12, characterized in that, Obtaining the failure probability values ​​of each computing device in the heterogeneous computing system that malfunctions and requires recalculation includes: Obtain the general failure probability of each computing device in the heterogeneous computing system; determine the historical failure probability of each computing device based on the historical failure data of each computing device in the heterogeneous computing system. The failure probability value of each computing device is determined based on the general failure probability and corresponding weight coefficient of each device, and the historical failure probability and corresponding weight coefficient of each device.

14. The fault handling method according to any one of claims 1 to 9, characterized in that, The process of acquiring computing resource information of the heterogeneous computing system includes: When a fault is detected in the computing device of the heterogeneous computing system during the execution of the inference request task, the fault device identification information corresponding to the fault device is obtained; the target processing stage of the inference request task at the time of the fault is obtained, and the task execution progress of the inference request task in the target processing stage is determined. Obtain the checkpoint backup data corresponding to the inference request task; The faulty device identification information, the target processing stage, the task execution progress, and the amount of backup data at the checkpoint are used as fault monitoring information.

15. The fault handling method according to claim 14, characterized in that, Determining the task execution progress of the inference request task in the target processing phase includes: Calculate the computational cost of each neural layer of the inference model corresponding to the inference request task in the target processing stage, and determine the total computational cost; The target computational load of the network neural layer already executed in the target processing stage is determined, and the task execution progress is determined based on the ratio of the target computational load to the total computational load.

16. A fault handling device, characterized in that, include: The data acquisition module is used to acquire inference task information and computing resource information of the heterogeneous computing system when a source computing device in the heterogeneous computing system fails during the execution of an inference request task. The inference task information includes resource requirement information and source device information, and the computing resource information includes performance parameters and fault monitoring information of each computing device. The source computing device is the computing device corresponding to the pre-filling stage and the decoding stage of the inference request task, including a fault-free computing device. The migration data determination module is used to determine, based on the inference task information and the computing power resource information, the estimated task duration required to continue executing the remaining tasks of the inference request task using a fault-free computing device in the heterogeneous computing system. The fault recovery module is used to select the target computing device with the lowest estimated task duration, generate a task recovery request that carries at least the target computing device identifier and the task migration instruction, and send it to the heterogeneous computing system to migrate the inference request task to the target computing device for continued execution. The migration data determination module is further configured to: designate computing devices in the heterogeneous computing system that are not the source computing devices, are not in a task execution state, and are in a normal state as candidate migration devices; and determine the task recovery parameters for each candidate migration device and the source computing device without faults to continue executing the inference request task based on the fault monitoring information and the resource requirement information, including: determining the network bandwidth between the faulty device in the source computing device and each candidate migration device and the source computing device without faults based on the performance parameters of each computing device; determining the amount of checkpoint backup data based on the fault monitoring information; and determining the data volume based on the data volume and... The network bandwidth between the source computing device and each candidate migration device and the fault-free source computing device is determined to calculate the migration cost of migrating the inference request task from the source computing device to each candidate migration device and the fault-free source computing device. Based on resource requirement information, the amount of data to be transmitted during the conversion process of the inference request task in the pre-filling and decoding stages is determined. Based on the performance parameters of each computing device, the network bandwidth between any two computing devices among the candidate migration devices and the fault-free source computing devices is determined. Based on the amount of data to be transmitted, the data transmission cost between any two computing devices among the candidate migration devices and the fault-free source computing devices is determined.

17. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the fault handling method as described in any one of claims 1 to 15.

18. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the fault handling method as described in any one of claims 1 to 15.

19. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the fault handling method according to any one of claims 1 to 15.

20. A heterogeneous computing system, characterized in that, It includes various types of computing devices and processors, with each type of computing device connected to the processor; The processor, when executing a computer program stored in the memory, implements the steps of the fault handling method as described in any one of claims 1 to 15.

Images