Method and device for self-recovery of computing power running task exception of intelligent computing cloud platform

Abstract

The application provides a computing power operation task exception self-recovery method and device of an intelligent computing cloud platform, relates to the technical fields of intelligent computing centers, intelligent computing centers, computing power infrastructure and intelligent computing cloud technology, and comprises the following steps: S1, collecting multi-dimensional runtime signals and computing resource state signals in the running process of a computing power operation task; S2, time-aligning the multi-dimensional runtime signals according to training step boundaries, and then decomposing the wall clock duration of each training step to obtain the time consumption of multiple mutually exclusive waiting categories; S3, determining the time consumption proportion of each waiting category in the wall clock duration, matching each time consumption proportion with the trigger condition of each candidate abnormal processing strategy to determine and execute a target abnormal processing strategy to obtain an execution result; and S4, determining a target fault type based on the computing resource state signals and the execution result, and executing a target recovery action corresponding to the target fault type to perform abnormal self-recovery. The application can solve the technical problems of difficult performance attribution and extensive self-recovery.

Description

Technical Field

[0001] This invention relates to the fields of intelligent computing centers, smart computing centers, computing infrastructure, and smart cloud technologies, specifically to a method and apparatus for self-healing abnormal computing tasks on an intelligent computing cloud platform. Background Technology

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "smart computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models) by utilizing large-scale heterogeneous computing resources, including general-purpose and intelligent computing power. Intelligent computing centers encompass facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0004] "Intelligent computing center" includes, but is not limited to, "intelligent computing center".

[0005] "Intelligent computing center" or artificial intelligence computing center is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting artificial intelligence computing architecture.

[0006] "Computing power" is the core of "intelligent computing center" and "smart computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to execute a certain computing requirement. It is the computing power to achieve the target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] With the advent of the era of large-scale models, the demand for computing power in distributed AI training has grown exponentially, leading to the widespread deployment of intelligent computing cloud platforms based on large-scale computing clusters. In existing large-scale distributed AI training technologies, to address frequent anomalies such as accelerator card failures, slow nodes, and network jitter, and to pinpoint the root causes of performance degradation such as decreased computing throughput, users are typically provided with a set of dispersed and independent resource utilization curves (such as GPU utilization and network bandwidth occupancy) for manual troubleshooting. The fault handling process employs a static, unified recovery model of "stopping the task—isolating the faulty node—replacing resources—overall recovery based on checkpoints."

[0008] However, this approach cannot accurately pinpoint the root cause of the slowdown in computing tasks (i.e., it cannot distinguish whether it is due to waiting for data loading, copying video memory, or ensemble communication). Furthermore, the performance bottleneck detection process and the fault self-healing process for computing tasks are disconnected, lacking a cascading solution from low-cost configuration optimization to high-cost hardware recovery. This makes it difficult for the intelligent computing cloud platform to determine the responsibility for the decline in computing task efficiency (whether it stems from user-side design flaws or platform-side defects). During the abnormal recovery phase of computing tasks, there is often over-handling that fails to differentiate the degree of fault impact (e.g., forcibly pausing the entire computing task and replacing the card for a "slow node" scenario with a relatively minor impact). Moreover, the static recovery strategy cannot be continuously adjusted based on the costs incurred in each actual handling (such as recovery time and computing power loss), resulting in excessively high fault recovery costs and a large amount of computing power that has not experienced any abnormalities remaining idle at the underlying level.

[0009] It is evident that since the emergence of intelligent computing centers, there has been a lack of effective solutions for performance attribution and fault self-healing of computing power operation tasks. This has resulted in the inability to accurately identify and automatically repair various anomalies in computing power operation tasks, and the inability to achieve a complete closed loop from performance attribution to fault self-healing. This has become a key bottleneck restricting the large-scale high-availability deployment of computing power operation tasks in intelligent computing cloud platforms. This problem is an urgent issue to be solved in this field. Summary of the Invention

[0010] This invention provides a method and apparatus for self-healing abnormalities in computing power operation tasks of an intelligent computing cloud platform, in order to solve the problem in the prior art that it is impossible to accurately define and automatically repair various abnormalities in computing power operation tasks and the complete closed loop from performance attribution to fault self-healing.

[0011] To solve the above problems, the present invention is implemented as follows: In a first aspect, the present invention provides a method for self-healing anomalies in computing power operation tasks on an intelligent computing cloud platform, comprising: Step S1: Collect multi-dimensional runtime signals and computing resource status signals generated by the model in the intelligent computing cloud platform during the execution of computing power tasks; Step S2: Align the multidimensional runtime signal according to the boundary of the training step to obtain the wall clock duration of each training step, and decompose the wall clock duration to determine multiple mutually exclusive waiting categories and the duration of the waiting category. Step S3: Determine the time consumption ratio of each waiting category in the wall clock duration, and match each time consumption ratio with the triggering conditions of each candidate exception handling strategy to determine the matching target exception handling strategy, and execute the target exception handling strategy to obtain the execution result; Step S4: Determine the target fault type based on the computing resource status signal and the execution result, determine the target recovery action corresponding to the target fault type, and execute the target recovery action to perform the abnormal self-healing of the computing power operation task.

[0012] In one embodiment, the multidimensional runtime signals include at least a data loading queue level signal, an inter-device data copy hardware counter signal, a computation kernel function execution timeline signal, and a collection communication event log signal. The multidimensional runtime signals are all continuously recorded throughout the entire process of the computing power operation task in the form of interval events carrying timestamps.

[0013] In one embodiment, step S2 includes: Step S2.1: Merge and sort all interval events recorded by the multidimensional runtime signal within the wall clock duration of the training step according to the start timestamp to obtain an event sequence on a unified time axis; Step S2.2: Identify the state transition points in the event sequence, wherein the state transition point is the moment when any execution state or waiting state ends and another state begins; Step S2.3: Based on the state transition points, divide the time axis of the training step into multiple state intervals that cover the entire training step interval, wherein no two state intervals overlap. Step S2.4: Determine the waiting category to which each state interval belongs, and sum the durations of all state intervals under the same waiting category to obtain the time consumption of each waiting category. The waiting category includes at least: waiting for data loading, waiting for data copying, waiting for calculation, and waiting for communication.

[0014] In one embodiment, the waiting category further includes waiting for slow nodes, which characterizes the additional waiting caused by nodes with abnormal computation time; step S2 further includes: Step S2.5: Obtain the computation time of each computing node participating in the training step; Step S2.6: Determine the median computation time of all the computing nodes; Step S2.7: Calculate the ratio of the computation time of each computing node to the median; Step S2.8: When the ratio of the target computing power node is continuously greater than the first preset threshold for more than a first preset number of training steps, the target computing power node is determined to be a slow node. Step S2.9: Include the portion of the computation time of the target computing node that exceeds the median into the time spent waiting for the slow node, and deduct the corresponding excess portion from the time spent waiting for computation.

[0015] In one embodiment, the candidate exception handling strategies and the triggering conditions corresponding to each candidate exception handling strategy are pre-configured in a strategy library; wherein step S3 includes: Step S3.1: Determine the second preset threshold and the second preset quantity corresponding to the different waiting categories of each trigger condition configuration from the strategy library; Step S3.2: When the time consumption ratio of the waiting category is greater than the second preset threshold for more than the second preset number of training steps in a row, it is determined that the corresponding triggering condition is met, and the associated candidate exception handling strategy is taken as the target exception handling strategy.

[0016] In one embodiment, step S3 further includes: Step S3.3: Determine the change in the proportion of each waiting category before and after applying the target anomaly handling strategy; Step S3.4: If the magnitude of the change meets the preset improvement conditions, then retain the configuration corresponding to the target anomaly handling strategy, and use the result of retaining the configuration as the execution result; Step S3.5: If the change magnitude does not meet the improvement conditions, roll back to the configuration before the target anomaly handling strategy was applied, and use the result of the rollback configuration as the execution result.

[0017] In one embodiment, the computing resource status signal includes a hardware status signal, a communication status signal, and a training status signal; wherein, step S4 includes: Step S4.1: If the hardware status signal and / or the communication status signal meet the preset hard fault criteria, then determine the target fault type based on the hardware status signal and / or the communication status signal. Step S4.2: If neither the hardware status signal nor the communication status signal satisfies the hard fault criterion, and the time consumption ratio of the target waiting category among the multiple waiting categories continues to be greater than the third preset threshold for more than a third preset number of training steps, and the corresponding target anomaly handling strategy is rolled back, then the target fault type is determined based on the computing resource status signal associated with the target waiting category.

[0018] In one embodiment, the fault types include software anomalies, single-card hardware failures, slow nodes, and link hotspots. A hierarchical recovery strategy table is pre-built to record the correspondence between the fault types and candidate recovery actions of different interruption levels, as well as the expected cost estimate corresponding to each candidate recovery action. Step S4 further includes: Step S4.3: According to the hierarchical recovery strategy table, determine the candidate recovery action with the smallest expected cost estimate from at least one candidate recovery action corresponding to the target fault type as the target recovery action.

[0019] In one embodiment, the candidate recovery action includes at least one of the following: If the fault type is a software-related exception, then the training process will be restarted in place. If the fault type is a single-card hardware fault, then isolate the faulty card, replace the card, and perform recovery based on checkpoints; If the fault type is the slow node, then without stopping the computing power operation task, the data parallel load share borne by the slow node will be dynamically offloaded to other computing power nodes. If the fault type is the link hotspot, then a communication ring topology reconfiguration is triggered to bypass the abnormal physical link; The interruption of the computing power operation task caused by the dynamic unloading and the communication ring topology reconstruction is lower than that caused by the in-situ restart of the training process. The interruption of the in-situ restart of the training process is lower than that caused by the isolation fault card, the replacement card and the recovery based on checkpoints.

[0020] In one embodiment, the method further includes: Step S5: Determine the actual cost of performing the target recovery action, and update the graded recovery strategy table online based on the actual cost.

[0021] In one embodiment, step S5 includes: Step S5.1: Obtain the actual cost generated by performing the target recovery action to process the target fault type, wherein the actual cost is determined based on the time consumed by the target recovery action and the amount of computational loss, and the amount of computational loss is determined based on the waiting time of slow nodes and the waiting time of communication for each training step during the period from the occurrence of the fault to the completion of recovery. Step S5.2: Based on a preset learning rate, the expected cost estimate of the target recovery action is weighted and summed with the actual cost to obtain an updated expected cost estimate; Step S5.3: Write the updated expected cost estimate back to the tiered recovery strategy table to update the tiered recovery strategy table.

[0022] In one embodiment, step S5 further includes: Step S5.4: Mark the occupancy status of the computing resources corresponding to the execution of the target recovery action as a specific status, so that the billing system can perform billing policy adjustment operations based on the specific status; Step S5.5: Determine the responsibility attribution based on the execution result of the target anomaly handling strategy; Step S5.6: Generate record information that records the target fault type, trigger time, target recovery action, recovery time, actual cost, and responsibility attribution determination result.

[0023] In one embodiment, step S5.5 includes: Step S5.5.1: If the execution result is to retain the configuration of the target exception handling strategy, then it is determined to be the user-side responsibility; Step S5.5.2: If the execution result is to roll back to the configuration before the application of the target exception handling strategy and trigger the determination of the target fault type, then it is determined to be the responsibility of the platform side.

[0024] In one embodiment, the method further includes: Step S6: After the target recovery action is completed, determine the change in the proportion of the waiting category associated with the target fault type before and after the fault recovery, and determine the fault recovery effect based on the change in the proportion information.

[0025] Secondly, the present invention also provides a self-healing device for abnormal computing power operation tasks of an intelligent computing cloud platform, comprising: The signal acquisition module is used to acquire multi-dimensional runtime signals and computing resource status signals generated by the model in the intelligent computing cloud platform during the execution of computing power tasks; The first determining module is used to time-align the multidimensional runtime signal according to the boundary of the training step to obtain the wall clock duration of each training step, and decompose the wall clock duration to determine multiple mutually exclusive waiting categories and the duration of the waiting category. The second determining module is used to determine the proportion of time spent by each waiting category in the wall clock duration, and match each proportion of time spent with the triggering conditions of each candidate exception handling strategy to determine the matching target exception handling strategy, and execute the target exception handling strategy to obtain the execution result. The third determining module is used to determine the target fault type based on the computing resource status signal and the execution result, determine the target recovery action corresponding to the target fault type, and execute the target recovery action to perform the abnormal self-healing of the computing power operation task.

[0026] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the self-healing method for abnormal computing power operation tasks of the intelligent computing cloud platform as described in the first aspect above.

[0027] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the self-healing method for abnormal operation of computing power tasks on an intelligent computing cloud platform as described in the first aspect above.

[0028] Fifthly, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the self-healing method for abnormal operation of computing power tasks on an intelligent computing cloud platform as described in the first aspect above.

[0029] In this invention, step S1 involves collecting multidimensional runtime signals and computing resource status signals generated by the model during the operation of computing power tasks in the intelligent computing cloud platform; step S2 involves aligning the multidimensional runtime signals according to the boundaries of training steps to obtain the wall clock duration of each training step, and decomposing the wall clock duration to determine multiple mutually exclusive waiting categories and the time consumption of each waiting category; step S3 involves determining the time consumption ratio of each waiting category in the wall clock duration, matching each time consumption ratio with the triggering conditions of each candidate anomaly handling strategy to determine the matching target anomaly handling strategy, and executing the target anomaly handling strategy to obtain the execution result; step S4 involves determining the target fault type based on the computing resource status signals and the execution result, determining the target recovery action corresponding to the target fault type, and executing the target recovery action to perform anomaly self-healing of the computing power operation task. In this way, by collecting multi-dimensional signals and performing mutual exclusion decomposition on a unified time axis, performance bottlenecks can be accurately quantified. Then, by combining the execution results of the anomaly handling strategy with the computing resource status signals, a complete closed loop from performance attribution to fault self-healing can be achieved. This can accurately identify and automatically repair various anomalies in computing power operation tasks without interrupting the main training process as much as possible, effectively reducing the cost of manual intervention and the loss of idle computing power. Attached Figure Description

[0030] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart of a self-healing method for abnormal computing power operation tasks on an intelligent computing cloud platform provided by the present invention; Figure 2 This is a schematic diagram of the four-layer runtime signal acquisition and mutual exclusion waiting category decomposition proposed in this invention; Figure 3aThis is a flowchart of the online learning closed-loop process for fault classification recovery decision and cost write-back proposed in this invention; Figure 3b This is a diagram illustrating the online learning effect proposed according to the present invention; Figure 4 This is the attribution diagnosis and graded self-healing flowchart proposed in this invention; Figure 5 This is a schematic diagram of the training performance attribution and fault self-healing system architecture proposed in this invention. Figure 6 This is a structural diagram of a self-healing device for abnormal computing power operation tasks on an intelligent computing cloud platform provided by the present invention. Figure 7 This is a structural diagram of an electronic device provided by the present invention. Detailed Implementation

[0032] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0033] The “computing power” mentioned in this invention refers to the ability of computer equipment or computing / data centers to process information. It is the ability of computer hardware and software to work together to perform a certain computing requirement. It is the computing power to achieve the target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0034] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0035] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capabilities of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capabilities.

[0036] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB), and the disaster recovery ratio is an important indicator of security and reliability.

[0037] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0038] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0039] The “computing power” mentioned in this invention includes: general computing power, intelligent computing power, and supercomputing power.

[0040] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0041] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0042] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0043] The "intelligent computing center" described in this invention refers to a facility that, through the use of large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), primarily provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0044] The "intelligent computing cloud platform" mentioned in this invention, abbreviated as "intelligent computing cloud", refers to a cloud computing platform that integrates hardware and software resources based on an intelligent computing center.

[0045] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0046] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0047] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0048] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0049] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0050] The "model" mentioned in this invention includes, but is not limited to, "large language model" and "multimodal large model".

[0051] The "large language model" mentioned in this invention refers to a large-scale language model (LLM), which is a language model with a large number of parameters. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0052] The “Multimodal Large Models” mentioned in this invention refer to models that combine multimodal information such as text, images, videos, and audio for training, including but not limited to multimodal large language models.

[0053] The "Token" mentioned in this invention refers to a basic unit of measurement when large language models or multimodal models process text, code, image descriptions, document content, or other serialized inputs and outputs. It is important to emphasize that different models use different tokenizers, and the same text may correspond to different numbers of tokens in different models.

[0054] The “computing power running task” mentioned in this invention refers to a specific workload or job executed on computing power resources that requires a certain amount of computing power support, usually involving complex data processing, numerical calculation, model training or simulation scenarios.

[0055] The "multidimensional runtime signal" mentioned in this invention refers to the underlying signal continuously collected across dimensions from the software layer to the hardware layer during the normal operation of model training, used to characterize the time consumption of various operations such as computing, communication, and data reading and writing.

[0056] The “computing resource status signal” mentioned in this invention refers to a set of monitoring indicators (e.g., hardware, communication and training status signals) that are continuously collected during the operation of computing power tasks to characterize whether there are any abnormalities in the underlying physical resources or network links.

[0057] The "training step" mentioned in this invention refers to an iterative cycle in which a complete forward propagation, backward propagation, and parameter update are completed during the model training process.

[0058] The "wall clock duration" mentioned in this invention refers to the absolute natural duration of a single training step from the occurrence of the initial operation to the completion of the final operation.

[0059] The "time alignment" mentioned in this invention refers to the process of mapping signal events from different software and hardware layers, each with its own independent timestamp, to the same time axis and the start and end boundaries of the training steps.

[0060] The "waiting category" mentioned in this invention refers to the category obtained by mutually exclusive division of the total training time according to the different working / suspended states of the system process (such as calculation, communication, data loading, etc.).

[0061] The "candidate anomaly handling strategy" described in this invention refers to a pre-set resource configuration or parameter adjustment method that can be applied without interrupting the main training process; it can also be called a "candidate prescription".

[0062] The "target fault type" mentioned in this invention refers to the specific anomaly classification (such as hardware fault, slow node, etc.) determined after multi-dimensional signal fusion diagnosis of the system.

[0063] The "target recovery action" described in this invention refers to a handling operation that the system automatically selects and executes to restore the normal operation of a task, based on a determined target fault type.

[0064] The “interval event” described in this invention refers to a recording unit with a clear start timestamp and end timestamp, which represents the duration of a specific operation (such as a data copy or a communication).

[0065] The "unified time axis" mentioned in this invention refers to a one-dimensional time reference line formed by integrating signals acquired asynchronously in multiple dimensions, with the same system clock as the reference.

[0066] The "state transition point" mentioned in this invention refers to the critical timestamp on a unified timeline when a certain execution action or waiting action ends and immediately enters another action.

[0067] The “state interval” mentioned in this invention refers to an absolute time segment whose internal state does not change, divided by two adjacent state transition points, and any two intervals are tightly spliced ​​together in time without overlap.

[0068] The "waiting for slow nodes" mentioned in this invention refers to the additional waiting time generated by the system during the synchronization phase caused by some computing nodes (slow nodes) in the cluster whose computing speed is significantly lower than the overall level.

[0069] The “computing power node” mentioned in this invention refers to the basic computing unit (usually referring to a specific computing device or Rank process) that participates in the distributed model computing power operation task.

[0070] The "strategy library" mentioned in this invention refers to a data set that is pre-configured and stores the mapping relationship between "triggering conditions" and "candidate exception handling strategies", and can also be called a "prescription library".

[0071] The "improvement conditions" mentioned in this invention refer to: preset quantitative standards for evaluating the effectiveness of strategy execution, such as a significant decrease in the percentage of specific waiting time compared to before application.

[0072] The "hard fault criterion" described in this invention refers to a judgment rule that clearly indicates that a substantial error (such as device disconnection or ECC uncorrectable error) has occurred in the underlying physical hardware or communication link and requires direct intervention for recovery.

[0073] The "hierarchical recovery strategy table" mentioned in this invention refers to a pre-configured data structure used to record the mapping relationship between fault types and recovery actions with different interruption levels, and to maintain the expected cost of the actions.

[0074] The “expected cost estimate” mentioned in this invention refers to a numerical indicator used to quantify and estimate the system time and computing power loss required after a certain recovery action is implemented. The smaller the value, the higher the priority.

[0075] The "lazy loading method" described in this invention refers to a loading mechanism that, when performing checkpoint recovery, does not read and load all data at once, but loads only the checkpoint fragments currently needed as required, in order to compress the recovery blocking time.

[0076] The "dynamic unloading" described in this invention refers to a technical means of transferring and redistributing the batch computing load (data parallel load share) originally planned to be undertaken by abnormal computing nodes to the remaining normal computing nodes in the cluster without stopping the current overall computing power operation tasks.

[0077] The "communication ring topology reconstruction" described in this invention refers to dynamically changing the collective communication logic or physical connection path between cluster nodes, thereby bypassing the recovery action of physical links with packet loss or latency anomalies.

[0078] The "actual cost" mentioned in this invention refers to the post-event quantitative assessment value obtained after each fault recovery action is completed, based on the actual wall clock duration consumed and the calculated amount of loss.

[0079] The “computational loss” mentioned in this invention refers to the computational time that is not converted into effective model parameter updates due to node waiting (slow nodes, communication anomalies) from the occurrence of the fault to the completion of recovery.

[0080] The "specific state" mentioned in this invention refers to a special internal marker attached to computing resources that are in the process of fault recovery, which is used to establish state isolation with external business systems.

[0081] The "billing strategy adjustment operation" described in this invention refers to the business logic such as suspending billing or waiving fees that is automatically triggered when the billing system identifies that a resource is in the aforementioned "specific state".

[0082] The “responsibility attribution determination” described in this invention refers to the process by which the system automatically determines whether the performance degradation is due to improper user use (such as “user-side responsibility” caused by code and data pipeline problems) or due to underlying platform anomalies (such as “platform-side responsibility” caused by hardware or network failures) based on the retention or rollback results of the policy.

[0083] The "proportion change information" mentioned in this invention refers to the difference or comparison result between the proportion of waiting time consumed by the waiting category and the proportion during the abnormal period, which is decomposed and statistically analyzed again after the recovery action is performed and the task resumes operation. This can be used to verify whether the recovery action has effectively eliminated the bottleneck.

[0084] It is important to emphasize that, in intelligent computing cloud platforms, the precise attribution of performance and automated fault self-healing for large-scale AI distributed computing tasks have always been challenging issues in this field. The core challenge lies in accurately quantifying the root causes of task slowdowns without modifying tenant training code, and in completing anomaly recovery with minimal computing power interruption. Currently, the industry faces these difficulties primarily due to the following technical challenges: I. Difficulty in accurately aligning multi-dimensional operating signals leads to difficulty in assigning responsibility for performance degradation. Distributed training of large models involves multiple asynchronous concurrent stages, including data loading, inter-device copying, underlying kernel function computation, and ensemble communication. In actual operation, computation and communication operations often highly overlap on the timeline, meaning computation and communication occur simultaneously. Existing monitoring methods typically only provide scattered resource utilization curves (such as individual GPU utilization or network bandwidth percentages), failing to eliminate the time overlap between underlying concurrent operations. Due to the lack of a unified timeline, current technologies struggle to fully and mutually exclusively decompose the actual wall clock duration. If performance degradation occurs, such as a decrease in computing throughput, it is often impossible to accurately determine whether the slowdown in model training is due to improper design of the tenant's training code or data pipeline, or defects in the platform infrastructure such as hardware, network, or scheduling. This can lead to disputes over responsibility for performance degradation and affect the efficiency of anomaly handling.

[0085] Second, the abnormal self-healing methods are crude, resulting in the cost of fault handling exceeding the degree of fault impact. Existing automated recovery solutions generally adopt a static, unified model of "stopping tasks—isolating nodes—replacing cards—overall recovery." This is suitable for hard faults such as device outages, but for anomalies such as "slow nodes" or minor network jitter, directly pausing the entire training cluster and performing a shutdown and card replacement would cause a large amount of underlying computing power that is not experiencing faults to passively enter an idle waiting state. This coarse-grained approach does not consider the different impacts of different fault types on computing power tasks, which may lead to situations where the cost of handling the fault exceeds the actual impact of the fault, resulting in significant loss of computing power and time.

[0086] Third, the handling strategies are rigid and lack the ability to continuously optimize based on actual costs. Existing fault handling logic largely relies on static rules preset by operations and maintenance personnel. This leads to a disconnect between the performance bottleneck detection process and subsequent fault handling procedures, lacking an upgrade path from low-cost configuration trial and error (such as dynamically adjusting the load, verifying and rolling back) to high-cost hardware recovery. Furthermore, static rules cannot detect and record the actual wall clock duration and lost computing power consumed in each actual fault handling. Because they cannot be updated and learned based on real operational data, the fault handling strategy will remain at the initial configuration level for a long time, unable to be continuously optimized as computing power scales up. This requires frequent updates and maintenance by operations and maintenance personnel, resulting in high long-term operational costs.

[0087] In summary, the coarse-grained monitoring and static recovery schemes used in existing technologies are difficult to apply in scenarios involving large-scale deployment of computing power tasks in intelligent computing cloud platforms. The industry urgently needs an intelligent solution that can accurately attribute causes and assign responsibilities, and can achieve hierarchical self-healing based on actual costs. This invention solves this pressing problem in the field of computing power.

[0088] Please see Figure 1 , Figure 1 This is a flowchart of a self-healing method for abnormal task operation on an intelligent computing cloud platform provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step S1: Collect multi-dimensional runtime signals and computing resource status signals generated by the model in the intelligent computing cloud platform during the execution of computing power tasks.

[0089] It should be noted that in the distributed training scenario of an intelligent computing cloud platform, the operational status of different computing nodes and the underlying cross-device interaction logic are quite complex. Traditional monitoring methods typically rely on single hardware alarms or scattered system logs, lacking the ability to globally perceive multi-dimensional operational status. In this approach, the system cannot accurately obtain the true state of computation, communication, and data flow, making fault handling quite difficult.

[0090] Therefore, it is urgent to comprehensively collect multi-dimensional runtime signals and computing resource status signals during the operation of computing power tasks, so as to provide data support for subsequent time consumption breakdown and anomaly location.

[0091] In one possible implementation, the specific content of the multidimensional runtime signals and computing resource status signals can be flexibly configured according to the application scenario, without any restrictions.

[0092] The beneficial technical effect achieved by step S1 is that by acquiring the multi-dimensional runtime signals and computing resource status signals generated during the operation of computing power tasks, a comprehensive data foundation can be provided for subsequent accurate time alignment and multi-source fault analysis.

[0093] Step S2: Align the multidimensional runtime signal according to the boundary of the training step to obtain the wall clock duration of each training step, and decompose the wall clock duration to determine multiple mutually exclusive waiting categories and the duration of each waiting category.

[0094] It should be noted that in large-scale computing tasks, asynchronous concurrent operations of the underlying hardware (such as overlapping execution of computation and communication) may cause various time-consuming processes to overlap on the timeline. Related technologies typically use the method of accumulating the duration of each operation when performing performance statistics, but this cannot eliminate the time redundancy caused by concurrent overlap. This results in statistical results that do not match the actual wall clock duration, making it impossible for the system to accurately identify the core bottleneck causing the task to slow down.

[0095] Therefore, it is urgent to precisely align the multidimensional runtime signals according to the boundaries of the training steps, and to decompose the wall clock duration into multiple waiting categories in a mutually exclusive manner, so as to provide reliable data support for subsequent evaluation of performance bottlenecks and determination of responsibility for performance degradation.

[0096] The beneficial technical effect achieved by step S2 is that by unifying the asynchronously parallel multidimensional signals into the same training step boundary, the time repetition statistical error caused by the overlapping execution of underlying hardware (such as parallel computation and communication) can be eliminated, and a complete and mutually exclusive quantitative decomposition of the training wall clock duration can be achieved, thereby accurately analyzing the specific performance bottleneck category.

[0097] Step S3: Determine the time consumption ratio of each waiting category in the wall clock duration, and match each time consumption ratio with the triggering conditions of each candidate exception handling strategy to determine the matching target exception handling strategy, and execute the target exception handling strategy to obtain the execution result.

[0098] It's important to note that when facing performance degradation phenomena such as decreased computing power throughput, the causes are often a mixture of improper user-side configuration and underlying platform failures. Traditional troubleshooting processes typically tend to directly trigger severe physical node isolation or restarts without differentiating the severity levels of the anomalies. This one-size-fits-all intervention approach leads to the over-handling of many minor degradation events, not only interrupting the main training process but also resulting in a significant amount of computing power being idle.

[0099] Therefore, it is urgent to dynamically match candidate exception handling strategies by combining the time consumption ratio of waiting categories and obtain the actual execution results, so as to provide technical support for accurately triggering platform-side fault determination and avoiding computing power waste.

[0100] The beneficial technical effects achieved by step S3 are: without interrupting the main training process, candidate exception handling strategies can be automatically issued and verified for specific waiting categories, thereby filtering out performance degradation problems caused by improper user-side configuration, and providing a clear basis for elimination for subsequent platform-side fault diagnosis.

[0101] Step S4: Determine the target fault type based on the computing resource status signal and execution result, determine the target recovery action corresponding to the target fault type, and execute the target recovery action to perform abnormal self-healing of the computing power operation task.

[0102] It should be noted that after confirming the existence of a genuine anomaly on the platform side, the degree of damage to the overall computing power operation varies greatly depending on the type of fault (such as a single card failure or individual slow nodes). Traditional self-healing solutions lack the ability to dynamically respond to different fault characteristics, causing the system to still incur high costs of restarting and full reloading when facing minor anomalies, resulting in ineffective loss of computing power.

[0103] Therefore, it is urgent to determine the target fault type by fusing computing resource status signals with the results of previous strategy execution, and to execute the corresponding target recovery actions as needed, so as to provide technical support for achieving high availability, non-stop anomaly self-healing, and ensuring the efficient flow of overall computing power tasks.

[0104] The beneficial technical effects achieved by step S4 are: it can effectively combine the computing resource status signals and the execution results of the target anomaly handling strategy to accurately identify the real fault type and automatically execute the corresponding target recovery action, which can effectively avoid over-processing caused by false alarms of a single signal and achieve efficient self-healing and rapid recovery of computing power operation tasks.

[0105] In this invention, step S1 involves collecting multidimensional runtime signals and computing resource status signals generated by the model during the operation of computing power tasks in the intelligent computing cloud platform; step S2 involves aligning the multidimensional runtime signals according to the boundaries of training steps to obtain the wall clock duration of each training step, and decomposing the wall clock duration to determine multiple mutually exclusive waiting categories and the time consumption of each waiting category; step S3 involves determining the time consumption ratio of each waiting category in the wall clock duration, matching each time consumption ratio with the triggering conditions of each candidate anomaly handling strategy to determine the matching target anomaly handling strategy, and executing the target anomaly handling strategy to obtain the execution result; step S4 involves determining the target fault type based on the computing resource status signals and the execution result, determining the target recovery action corresponding to the target fault type, and executing the target recovery action to perform anomaly self-healing of the computing power operation task. In this way, by collecting multi-dimensional signals and performing mutual exclusion decomposition on a unified time axis, performance bottlenecks can be accurately quantified. Then, by combining the execution results of the anomaly handling strategy with the computing resource status signals, a complete closed loop from performance attribution to fault self-healing can be achieved. This can accurately identify and automatically repair various anomalies in computing power operation tasks without interrupting the main training process as much as possible, effectively reducing the cost of manual intervention and the loss of idle computing power.

[0106] In one embodiment, the multidimensional runtime signals include at least a data loading queue level signal, an inter-device data copy hardware counter signal, a computation kernel function execution timeline signal, and a collection communication event log signal. The multidimensional runtime signals are all continuously recorded throughout the entire process of the computing power operation task in the form of interval events carrying timestamps.

[0107] For example, in this embodiment of the invention, runtime signals can be collected through the following four levels without modifying the user's training code: (1) DataLoader queue level signal (i.e., the data loading queue level signal mentioned above): reflects whether the data preparation speed can keep up with the training consumption speed. It is obtained by monitoring the change of the number of elements in the output queue of the data loading process over time. (2) PCIe / NVLink hardware counter signal (i.e. the above-mentioned hardware counter signal for data copying between devices): reflects the time taken for data copying operations across devices (such as from host memory to video memory, between cards), and is obtained by reading the start and end timestamps of the copy interval exposed by the device driver or hardware performance counter; (3) CUDA kernel execution timeline signal (i.e. the above-mentioned computation kernel function execution timeline signal): reflects the actual computation time, obtained through the kernel startup and termination timestamps exposed by the underlying computation library or performance profiling interface; (4) NCCL communication event log signal (i.e. the above collection communication event log signal): reflects the time consumed by collection communication operations (such as AllReduce, AllGather), which is obtained by intercepting or subscribing to the start and end timestamps of communication events exposed by the collection communication library.

[0108] The above four layers of signals are all recorded in the form of time-stamped interval events (such as "a DataLoader dequeue waiting interval", "a PCIe copy interval", "a CUDA kernel execution interval", "a NCCL communication interval"), thus providing the original input for subsequent alignment by training step boundaries and decomposition into mutual exclusion waiting categories.

[0109] It should be noted that in this embodiment, the above four layers of signals are continuously collected throughout the entire process of the computing power operation task, and the waiting category decomposition is executed step by step according to the training steps; that is, signal collection and attribution are not based on the premise of determining the occurrence of a fault, but are normal operations, and the waiting category ratios output step by step are also used as one of the inputs for determining the fault type of the self-healing stage.

[0110] The beneficial technical effects achieved by this embodiment are as follows: by limiting the specific types of multidimensional runtime signals and the continuous recording method, it can be ensured that the collected data can fully cover the core links of large model training such as data loading, inter-card transmission, calculation and network communication, thereby improving the completeness of the collection of underlying runtime characteristics.

[0111] In one embodiment, step S2 includes: Step S2.1: Merge and sort all interval events recorded by the multidimensional runtime signal within the wall clock duration of the training step according to the start timestamp to obtain the event sequence on a unified time axis; Step S2.2: Identify state transition points in the event sequence, where a state transition point is the moment when any execution state or waiting state ends and another state begins; Step S2.3: Based on the state transition points, divide the time axis of the training step into multiple state intervals that cover the entire training step interval, wherein no two state intervals overlap. Step S2.4: Determine the waiting category to which each state interval belongs, and sum the durations of all state intervals under the same waiting category to obtain the time consumed by each waiting category. The waiting categories include at least: waiting for data loading, waiting for data copying, waiting for calculation, and waiting for communication.

[0112] The beneficial technical effects achieved by steps S2.1 to S2.4 are as follows: by merging sorting and identifying state transition points, the time axis of the training step can be divided into non-overlapping state intervals, so as to effectively avoid repeated calculation time consumption, ensure that the sum of the waiting time of each decomposed waiting category is equal to the total wall clock time, and improve the accuracy and reliability of the attribution calculation results.

[0113] In one embodiment, the wait category further includes waiting for slow nodes, which characterizes the additional wait caused by nodes with abnormal computation time; step S2 further includes: Step S2.5: Obtain the computation time of each computing node participating in the training step; Step S2.6: Determine the median computation time of all computing nodes; Step S2.7: Calculate the ratio of the computation time of each computing node to the median; Step S2.8: When the ratio of the target computing power node is continuously greater than the first preset threshold for more than a first preset number of training steps, the target computing power node is determined to be a slow node. Step S2.9: Include the portion of the computation time of the target computing node that exceeds the median into the time spent waiting for the slow node, and deduct the corresponding excess portion from the time spent waiting for computation.

[0114] The beneficial technical effects achieved by steps S2.5 to S2.9 are as follows: By combining the median and ratio of the computation time of each computing node, the extra waiting time caused by the slowdown of computing nodes (slow nodes) can be accurately separated from the normal computation time, which can effectively improve the accuracy of slow node identification.

[0115] In one possible implementation, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the four-layer runtime signal acquisition and mutual exclusion waiting category decomposition proposed according to the present invention. When converting the signals acquired independently from the four layers into complete and mutually exclusive quantitative attributions of the wall clock time in a single training step, it can be based on the following formal decomposition model: Let the total duration of the wall clock in the s-th training step be T_wall(s). This invention decomposes it into: T_wall(s) = T_data(s) + T_copy(s) + T_compute(s) + T_comm(s) + T_straggler(s) Where T_data(s), T_copy(s), T_compute(s), T_comm(s), and T_straggler(s) represent the time spent in each of the five waiting categories within this step: waiting for data loading, waiting for data copying, waiting for computation, waiting for communication, and waiting for slow nodes, respectively. These components are mutually exclusive, meaning that the training process can only be in one of the above five states at any given time.

[0116] To ensure that the decomposition satisfies both completeness (the sum of all components is strictly equal to T_wall(s)) and mutual exclusion (no overlap between any two components), this embodiment processes the four-layer signal according to the following steps: All timestamped interval events recorded by the four-layer signal within the step are merged and sorted according to their starting timestamps to obtain an event sequence on a unified timeline; state transition points (i.e., the moment when one waiting / execution state ends and another state begins) are identified sequentially along this event sequence; the entire timeline [start time of the step, end time of the step] is divided into several non-overlapping state intervals whose union covers the entire step interval according to the state transition points; each state interval is assigned to its corresponding component according to its waiting category, and the duration of all intervals of the same category is accumulated to obtain the four components T_data(s), T_copy(s), T_compute(s), and T_comm(s) of the step. Since the division of state intervals ensures coverage and non-overlap, the above decomposition naturally satisfies the completeness constraint and mutual exclusion constraint Σ = T_wall(s).

[0117] The method for determining T_straggler(s) is as follows: For all computing nodes (rank) participating in this step, record their computation time t_i(s) for this step (i is the rank number), calculate the median_j(t_j(s)) of the computation time of all ranks, and define the relative ratio of each rank: ratio_i(s) = t_i(s) / median_j(t_j(s)) When the ratio_i(s) of a certain rank i exceeds a preset threshold τ (typically 1.5), and this phenomenon occurs continuously for K consecutive training steps (K is a preset positive integer used to avoid misjudgments caused by momentary jitter), the rank is determined to be a slow node; the portion of the rank calculation time that exceeds the median (i.e., t_i(s)) is considered slow. The duration corresponding to median_j(t_j(s)) is included in T_straggler(s) of this step and is not counted again in T_compute(s).

[0118] For example: Suppose that the computation times for the four rank calculations within a certain step are t_1=210ms, t_2=205ms, t_3=208ms, and t_4=340ms, with a median of 209ms (the average of the two middle values, 208 and 210). Then, ratio_4 = 340 / 209 ≈ 1.63, exceeding the threshold τ=1.5. If this phenomenon persists for K=3 consecutive steps, then rank4 is determined to be a slow node, and the portion of its computation time exceeding the median of 340ms is considered slow. 209 = 131ms is included in T_straggler(s) for this step, and the remaining 209ms is included in T_compute(s).

[0119] In one embodiment, candidate exception handling strategies and corresponding triggering conditions are pre-configured in a strategy library; wherein step S3 includes: Step S3.1: Determine the second preset threshold and the second preset quantity corresponding to the different waiting categories configured for each trigger condition from the policy library; Step S3.2: When the time consumption ratio of the waiting category is greater than the second preset threshold for more than a second preset number of training steps, it is determined that the corresponding triggering condition is met, and the associated candidate exception handling strategy is taken as the target exception handling strategy.

[0120] For example, in this embodiment of the invention, after obtaining the time consumption corresponding to different waiting categories, a waiting category proportion vector for that step can be defined: p(s) = ( T_data(s), T_copy(s), T_compute(s), T_comm(s), T_straggler(s) ) / T_wall(s) The prescription library (i.e., the aforementioned strategy library) pre-defines several "trigger condition - candidate prescription (i.e., the aforementioned candidate exception handling strategy)" rules, which may include, for example: (1) Triggering condition: The proportion of T_data exceeds the preset threshold for N consecutive steps; Candidate prescription: Increase the number of DataLoaderworkers / Enable data prefetching.

[0121] (2) Triggering condition: The proportion of T_copy exceeds the preset threshold for N consecutive steps; Candidate prescription: Adjust the data copying method or enable pinned memory.

[0122] (3) Triggering condition: The proportion of T_straggler exceeds the preset threshold for N consecutive steps; Candidate prescription: Rebalance the data parallel batch share undertaken by each computing node.

[0123] (4) Triggering condition: The proportion of T_comm exceeds the preset threshold for N consecutive steps; Candidate prescription: Adjust the size of the gradient communication bucket or calculate the communication overlap strategy.

[0124] The beneficial technical effects achieved by steps S3.1 to S3.2 are as follows: by setting the second preset quantity and the second preset threshold as the triggering conditions of the time dimension and the proportion dimension, respectively, the normal performance fluctuations in the operation of computing power running tasks can be effectively filtered, the invalid triggering of the abnormal handling strategy can be avoided, and the stability of the self-healing process can be improved.

[0125] In one embodiment, step S3 further includes: Step S3.3: Determine the change in the proportion of each waiting category before and after applying the target exception handling strategy; Step S3.4: If the magnitude of change meets the preset improvement conditions, retain the configuration corresponding to the target anomaly handling strategy and use the result of retaining the configuration as the execution result. Step S3.5: If the change magnitude does not meet the improvement conditions, roll back to the configuration before the application of the target exception handling strategy, and use the result of the rollback configuration as the execution result.

[0126] In one possible implementation, when the proportion vector p(s) obtained in the above embodiments meets the triggering condition of a certain rule within N consecutive steps, the system can automatically apply the candidate prescription corresponding to the rule without affecting the main training progress (e.g., using checkpoint gaps or shadow processes), and respectively count the changes in the proportion of the waiting category before and after the application of the prescription: if the change meets the preset significant improvement condition (e.g., the proportion of the target waiting category decreases by more than a preset amount), the configuration corresponding to the prescription is retained; if no significant improvement is achieved or deterioration occurs, the configuration before the prescription is applied is rolled back. This mechanism can achieve a complete closed loop of "automatic handling - effect verification - retention or rollback" after the bottleneck category is detected.

[0127] The beneficial technical effects achieved by steps S3.3 to S3.5 are as follows: By comparing the changes before and after the application strategy to achieve closed-loop verification, it can be ensured that invalid or performance-degrading configurations can be rolled back in a timely manner, avoiding inappropriate adjustments from causing continuous negative impacts on computing power operation tasks.

[0128] In one embodiment, the computing resource status signal includes a hardware status signal, a communication status signal, and a training status signal; wherein, step S4 includes: Step S4.1: If the hardware status signal and / or communication status signal meet the preset hard fault criteria, then determine the target fault type based on the hardware status signal and / or communication status signal. Step S4.2: If neither the hardware status signal nor the communication status signal meets the hard fault criterion, and the time consumption ratio of the target waiting category in multiple waiting categories continues to be greater than the third preset threshold for more than a third preset number of training steps, and the corresponding target anomaly handling strategy is rolled back, then the target fault type is determined based on the computing resource status signal associated with the target waiting category.

[0129] The beneficial technical effects achieved by steps S4.1 to S4.2 are: clear hard faults can be dealt with directly to shorten the response time; for implicit performance degradation, the results of the strategy rollback are analyzed to effectively distinguish the boundaries of software and hardware anomalies and prevent misjudgment or omission of fault types.

[0130] In one possible implementation, the self-healing segment determines the type of fault that occurs in the computing power operation task based on the following three types of status signals continuously collected during the operation of the computing power operation task: hardware status signals (such as device-level indicators like ECC error count and GPU XID error codes), communication status signals (such as communication layer indicators like NCCL communication timeout and message retransmission), and training status signals (such as training layer indicators like loss value and gradient norm). It should be noted that these status signals are continuously collected during the normal operation of the computing power operation task; a fault is not a prerequisite for collection. A fault is a judgment result obtained after applying a preset anomaly criterion to the above signals.

[0131] In one possible implementation, the fault type is determined using a dual-entry mechanism: Straight-through entry: When hardware or communication status signals meet preset hard fault criteria (e.g., ECC uncorrectable error count greater than zero, GPU XID offline error code, or aggregated communication operation timeout), the fault type (e.g., single-card hardware fault, software anomaly) is directly determined based on the signal combination characteristics without needing to go through the prescription trial in the attribution stage, and the tiered recovery process begins. In hard fault scenarios, there are no candidate prescriptions available for trial, and the straight-through entry avoids processing delays.

[0132] Upgrade Entry Point: When the proportion of a certain waiting category in the attribution segment continuously exceeds a preset threshold, and the matched candidate prescription is rolled back after verification, this rollback event triggers an upgrade decision. The system determines the fault type by combining the status signal corresponding to that waiting category. For example: (1) T_straggler proportion exceeds threshold, candidate prescription "rebalancing data parallel batch share" is rolled back: If the hardware status signal of the slow rank is normal (no ECC / XID abnormality, no temperature or frequency abnormality), it is determined to be a "slow node", and the corresponding recovery action is to dynamically unload its load share; if the hardware status signal of the rank is abnormal (such as ECC correctable error count continuously increasing, temperature or frequency abnormality), it is determined to be a "single card hardware failure", and the corresponding recovery action is to isolate and replace the card.

[0133] (2) T_comm proportion exceeds the threshold, candidate prescription "adjust communication bucket size or calculate - communication overlap strategy" is rolled back: if the communication status signal shows that the retransmission / packet loss count of a specific physical link is abnormal, it is determined to be "link hot spot", and the corresponding recovery action is communication ring topology reconstruction; if the communication status signal shows that the aggregate communication operation repeatedly times out and there is no specific link abnormality, it is determined to be "software abnormality", and the corresponding recovery action is to restart the training process in place.

[0134] (3) If the proportion of T_data or T_copy exceeds the threshold and the candidate prescription is rolled back, but there is no corresponding abnormal status signal on the platform side: it is not upgraded to a fault type, and the attribution conclusion is recorded as a user-side data pipeline design problem, which is directly used for subsequent responsibility determination and billing dispute handling.

[0135] Therefore, "grading" has a clear meaning in this invention: Level 0 is a candidate prescription that does not interrupt training and does not occupy additional hardware resources; Level 1 is dynamic offloading of slow node load and communication ring topology reconstruction without stopping computing power operation tasks; Level 2 is restarting the training process in place; Level 3 is isolating faulty cards, supplementing replacement cards, and recovering based on checkpoints. The degree of interruption to computing power operation tasks increases sequentially at each level. For performance degradation anomalies, the system starts handling from Level 0 and only escalates when lower-level handling is verified to be ineffective; for anomalies that meet the hard fault criteria, they directly enter the corresponding level through a direct entry point.

[0136] In one embodiment, the fault types include software anomalies, single-card hardware failures, slow nodes, and link hotspots. A hierarchical recovery strategy table is pre-built to record the correspondence between fault types and candidate recovery actions of different interruption levels, as well as the expected cost estimate for each candidate recovery action. Step S4 further includes: Step S4.3: Based on the graded recovery strategy table, determine the candidate recovery action with the smallest expected cost estimate from at least one candidate recovery action corresponding to the target fault type as the target recovery action.

[0137] The beneficial technical effect achieved by step S4.3 is that by combining the hierarchical recovery strategy table with the expected cost estimate, the system can be guided to prioritize the action with the least impact on the overall task and the lowest historical execution cost when facing different faults, so as to reduce the computing power loss caused by fault handling.

[0138] In one possible implementation, let the fault type set F = {software anomalies, single-card hardware failures, slow nodes, link hotspots}. Based on the combined characteristics of the above three types of status signals, the waiting category proportion vector output by the attribution segment, and the candidate prescription verification results, the system maps the current anomaly to one or more fault types in F, which serve as input for subsequent queries of the graded recovery strategy table. The determination of "slow nodes" reuses the determination method based on ratio_i(s) and threshold τ in the above embodiments, realizing the reuse of capabilities between the attribution segment and the self-healing segment in the slow node detection stage. The "anomaly" referred to in this invention encompasses two categories: performance degradation anomalies (such as a waiting category proportion continuously exceeding a threshold) and fault anomalies. The "fault type" is the classification result of the anomaly after being determined by the above criteria. The "anomaly self-healing" in the name refers to the automatic handling of the above two types of anomalies using candidate prescriptions and graded recovery actions, respectively.

[0139] To ensure that the above three types of state signals can be stably and non-intrusively acquired by the system, the present invention specifies the following limitations on the specific acquisition methods for each type of signal: (1) Hardware status signals: By periodically querying the error count register and error code field exposed by the device management interface provided by the accelerator card manufacturer (such as NVIDIA DCGM / NVML, or the management interface corresponding to similar domestic accelerator cards), device-level abnormal indicators such as ECC uncorrectable error count and GPU XID error code can be obtained. (2) Communication status signals: By subscribing to or intercepting the timeout callbacks and retransmission / packet loss counters exposed by the aggregated communication library (such as NCCL), communication layer abnormal indicators such as communication operation timeout and message retransmission are obtained. This collection channel shares the same underlying interception mechanism as the above-mentioned aggregated communication event log signals. (3) Training status signals: Without modifying the user's training code, by attaching a lightweight monitoring hook (such as a callback interface after backpropagation is completed) to the loss calculation and gradient calculation stages of the training framework, the loss value and gradient norm of each training step are read, and abnormal indicators of the training layer such as sudden changes in loss value and abnormal gradient norm are obtained.

[0140] The acquisition methods for the three types of status signals mentioned above are consistent with those for the four-layer runtime signals. All three types are acquired by listening to the exposed interfaces / callbacks of the device driver, communication library, or training framework in read-only mode. This does not intrude on or modify the computational logic of the user's training code, nor does it rely on customized modifications to the training framework. The three types of status signals are continuously acquired throughout the training process using three rhythms: periodic polling (hardware status signals, typically with a period on the order of seconds), event callbacks (communication status signals), and step-by-step hooks (training status signals). The acquisition overhead is low, and these signals can be enabled normally.

[0141] In one embodiment, the candidate recovery action includes at least one of the following: If the fault type is a software exception, then the training process will be restarted in place. If the fault type is a single card hardware fault, then isolate the faulty card, replace it with a replacement card, and perform recovery based on checkpoints; If the fault type is a slow node, the data parallel load share borne by the slow node will be dynamically offloaded to other computing nodes without stopping the computing power to run tasks. If the fault type is a link hotspot, then a communication ring topology reconfiguration is triggered to bypass the abnormal physical link; The interruption of the computing power operation task caused by the dynamic unloading and the communication ring topology reconstruction is lower than that caused by the in-situ restart of the training process. The interruption of the in-situ restart of the training process is lower than that caused by the isolation fault card, the replacement card and the recovery based on checkpoints.

[0142] The beneficial technical effect achieved by this embodiment is that by limiting specific recovery actions (such as lazy loading, dynamic load unloading, communication ring topology reconstruction, etc.) with different interruption levels for different specific fault types, the continuous operation of the original computing power tasks can be maintained as much as possible while ensuring that the fault is effectively isolated.

[0143] In other words, in this embodiment of the invention, for each fault type f, a corresponding set of candidate recovery actions A_f can be pre-defined. Each recovery action is classified according to its degree of interruption to the computing power running task, and a graded recovery strategy table records the correspondence between fault types and recovery actions of different interruption degrees. A specific embodiment of the invention exemplarily sets: Software exceptions correspond to {restarting the training process in place}: These exceptions usually originate from transient errors in the software layer, and can be recovered by restarting the process without replacing hardware resources; For single-card hardware failure, the corresponding candidate actions are {isolation and card replacement followed by checkpoint-based lazy loading recovery and in-place retry}: Isolate the faulty card, replace the card, and then perform recovery based on checkpoints. Checkpoint recovery can adopt a segmented checkpoint lazy loading method, loading checkpoint segments as needed instead of loading all data at once, so as to compress the recovery time required. Slow nodes correspond to {dynamic offloading of data parallel load share}: without stopping the entire computing power operation task, only the data parallel load share borne by the slow node is dynamically offloaded to other computing power nodes, so that the computing power operation task can continue to run at near normal throughput before the slow node problem is resolved, which is different from the existing technology's "shutdown and card replacement" approach. Link hotspot corresponds to {communication ring topology reconfiguration}: triggers communication ring topology reconfiguration to bypass physical links experiencing abnormal delays.

[0144] In one embodiment, the method further includes: Step S5: Determine the actual cost of performing the target recovery action, and update the graded recovery strategy table online based on the actual cost.

[0145] The beneficial technical effect achieved by step S5 is that by updating the hierarchical recovery strategy table online based on actual costs, the hierarchical recovery strategy can be continuously optimized as operational data accumulates, thereby improving the practicality of the hierarchical recovery strategy table.

[0146] In one embodiment, step S5 includes: Step S5.1: Obtain the actual cost generated by executing the target recovery action to handle the target fault type. The actual cost is determined based on the time consumed by the target recovery action and the amount of computational loss. The amount of computational loss is determined based on the waiting time of the slow node and the waiting time of communication for each training step during the period from the occurrence of the fault to the completion of the recovery. Step S5.2: Based on the preset learning rate, the expected cost estimate of the target recovery action is weighted and summed with the actual cost to obtain the updated expected cost estimate; Step S5.3: Write the updated expected cost estimate back to the tiered recovery strategy table to update the tiered recovery strategy table.

[0147] The beneficial technical effects achieved by steps S5.1 to S5.3 are as follows: By adopting a weighted summation method based on the learning rate, the expected cost estimate in the hierarchical recovery strategy table can be continuously and dynamically updated and corrected as actual operating data accumulates. This ensures that the system prioritizes operations with lower actual recovery costs when dealing with similar faults in the future, thereby improving the rationality of decision-making in the long-term operation and maintenance of the intelligent computing cloud platform.

[0148] In one possible implementation, such as Figure 3a As shown, Figure 3aThis invention presents a closed-loop flowchart for online learning of fault classification recovery decision-making and cost write-back, where a formal online learning model can be defined as follows: A classification recovery strategy table maintains the expected cost estimate E[c(f,a)] for each (f∈F, fault type, a∈A_f, candidate recovery action). The cost c is determined by combining the time consumed in recovering the fault and the computational cost lost due to the fault. The recovery time is the wall clock duration from determining the fault type to the completion of the recovery action and the resumption of normal iteration of the computing power task. The lost computational cost is determined based on the waiting category decomposition results of each training step in the attribution segment, i.e., the sum of the waiting time T_straggler and the waiting communication time T_comm for each training step from the occurrence of the fault (or the triggering of the upgrade entry) to the completion of recovery (for hard faults handled through the direct entry, it is the computational cost converted from the wall clock duration during task interruption). After the system actually executes recovery action a to handle fault f, it records the actual cost c_obs and updates the expected cost estimate using the following moving average method: E[c(f,a)] ← (1 α)·E[c(f,a)] + α·c_obs Where α is the preset learning rate (0 < α < 1). When a similar fault f is encountered again, the system prioritizes the action with the smallest E[c(f,a)] from the candidate recovery action set A_f, so that the hierarchical recovery strategy table continuously optimizes the decision based on actual operating data, rather than remaining fixed at the initial configuration.

[0149] like Figure 3b As shown, Figure 3b This is an online learning effect diagram proposed according to the present invention. Numerical example: Suppose the fault type f = single card hardware fault, the recovery action a = lazy loading recovery after isolation and card replacement, and its expected cost estimate initial value E[c(f,a)] = 120 (relative cost score, calculated by combining time consumption and lost computing power), and the learning rate α = 0.2. After a certain actual execution of this recovery action, the actual cost c_obs = 90 is observed (due to the superimposed lazy loading mechanism, the recovery time is lower than the historical average), then the updated E[c(f,a)] ← (1 0.2)×120 + 0.2×90 = 96 + 18 = 114. After multiple instances where the actual execution cost is consistently lower than the historical estimate, E[c(f,a)] will gradually converge to a level closer to the actual cost (e.g., ...). Figure 3b As shown in the example table, the E[c] of the (f,a) pair is updated from an initial value of 120 to 96 through online learning; conversely, if the actual cost of a candidate action (such as "retrying in place" corresponding to a single-card hardware failure) is higher than the historical estimate in each iteration, its E[c] will gradually increase with online learning (e.g., ...). Figure 3b(Updated from 150 to 162 in the example), thereby reducing the probability of this action being prioritized in subsequent decisions. This numerical example illustrates the core effect of the online learning mechanism: "actions with lower actual costs are prioritized."

[0150] Recovery effect verification: After the recovery action is completed, the attribution segment continues to perform waiting category decomposition according to the training step boundary. The system compares the waiting category proportion vector p(s) before and after recovery: if the proportion of the waiting category that triggered this action falls below the preset threshold, the recovery is confirmed to be effective, and the actual cost c_obs is recorded as described above; if it does not fall, the recovery action is determined to be invalid, and c_obs is recorded according to the preset cost upper limit of the action, and the action with the second smallest expected cost is selected from the candidate recovery action set A_f for further processing. This verification method allows the effect evaluation of the self-healing segment to share the same decomposition model with the attribution segment, eliminating the need to deploy separate evaluation methods.

[0151] In one embodiment, step S5 further includes: Step S5.4: Mark the occupancy status of the computing resources corresponding to the target recovery action as a specific status, so that the billing system can perform billing policy adjustment operations based on the specific status; Step S5.5: Determine the responsibility attribution based on the execution results of the target anomaly handling strategy; Step S5.6: Generate record information including the target fault type, trigger time, target recovery action, recovery time, actual cost, and responsibility attribution determination result.

[0152] The beneficial technical effects achieved by steps S5.4 to S5.6 are as follows: By automatically marking specific resource states and linking them to billing adjustments, the risk of overcharging users due to platform failures is eliminated, and the generated record information can meet the needs of subsequent operation and maintenance audits.

[0153] In one embodiment, step S5.5 includes: Step S5.5.1: If the execution result is to retain the configuration of the target exception handling strategy, then it is determined to be the user's responsibility. Step S5.5.2: If the execution result is to roll back to the configuration before the application target exception handling strategy and trigger the determination of the target fault type, then it is determined to be the responsibility of the platform side.

[0154] In other words, in this embodiment, the verification results of the aforementioned candidate prescriptions, in addition to determining whether to retain or roll back, can also serve as input for determining the fault type in the self-healing phase: if a candidate prescription is verified to be retained, it indicates that the performance degradation corresponding to the waiting category can be eliminated through adjustments on the user side / configuration side, and the attribution conclusion is recorded as user-side responsibility; if a candidate prescription is verified to be rolled back (i.e., adjustments to the user-side configuration cannot improve the situation), and the proportion of the waiting category continues to exceed the threshold, it indicates that the performance degradation originates from abnormalities in platform-side resources, links, or nodes. In this case, the rollback event triggers the fault type determination process in the self-healing phase, and the attribution conclusion is recorded as platform-side responsibility. Thus, the candidate prescriptions constitute the Level 0 handling in the hierarchical handling of this invention—not interrupting training and not occupying additional hardware resources, and their verification results determine whether to upgrade to a higher-level recovery action.

[0155] The beneficial technical effects achieved by steps S5.5.1 to S5.5.2 are: the final execution result of the exception handling strategy can be used to reverse-engineer the objective division of the responsibility boundary between the platform side and the user side, and provide data support for subsequent billing dispute handling and operation and maintenance optimization.

[0156] In one embodiment, the method further includes: Step S6: After the target recovery action is completed, determine the change in the proportion of waiting time associated with the target fault type before and after the fault recovery, and determine the fault recovery effect based on the proportion change information.

[0157] The beneficial technical effect achieved by step S6 is that by comparing the changes in the time consumption ratio of relevant waiting categories before and after fault recovery after the target recovery action is performed, the fault recovery effect can be objectively verified.

[0158] In other words, in this embodiment of the invention, the system can mark the occupancy status of the corresponding computing resources as a specific state (e.g., "fault recovery in progress") according to the fault recovery process. This specific state is used to associate with the billing system's billing strategy adjustment, enabling the billing system to automatically identify whether the resources are in a normal and available state during the period based on the resource occupancy status, without the need for manual verification and intervention. Simultaneously, the system generates records of the fault type, trigger time, recovery actions taken, recovery time, and actual cost, forming a traceable and auditable "fault-recovery-cost" record link for subsequent responsibility determination and strategy optimization analysis. Responsibility attribution determination: When a candidate prescription in the attribution segment is verified and retained, the system records the responsibility for this performance degradation as the user side (training code or data pipeline configuration); when the candidate prescription is rolled back and determined to be a fault type through the upgrade entry in the above embodiment, it is recorded as the platform side (hardware, link, or scheduling). The above responsibility attribution and the corresponding training step's waiting category percentage evidence are written into an auditable record for use in billing dispute resolution.

[0159] In summary, as described in the above embodiments, Figure 4 As shown, Figure 4 This invention presents a flowchart for attribution diagnosis and graded self-healing, which can be divided into two stages: an attribution stage and a self-healing stage. The attribution section sequentially performs four layers of signal acquisition, boundary alignment according to training steps, mutual exclusion wait category decomposition, wait category proportion matching with prescription library, automatic application of candidate prescriptions and short-term comparison verification. The self-healing phase sequentially performs multi-source state signal fusion and classification, determines and executes recovery actions according to the hierarchical recovery strategy table, records the actual cost, and writes back and updates the hierarchical recovery strategy table in a moving average manner. At the same time, it marks the resource occupancy status during the recovery process as a specific status and generates auditable records.

[0160] The two pipelines share the same computing power to run parallel topology information and step boundary information, and there is bidirectional data coupling: First, the waiting category proportion vector output by the attribution segment and the verification results of the candidate prescriptions serve as inputs for determining the fault type of the self-healing segment. The rollback of the candidate prescriptions constitutes the upgrade trigger condition of the self-healing segment. Secondly, the calculation of the loss in the actual cost of the self-healing recovery action and the verification of the recovery effect are both determined based on the waiting category decomposition results of the training step by step in the attribution phase.

[0161] This forms a complete closed loop from attributing performance bottlenecks to fault self-healing, and then measuring the self-healing effect using the attribution model.

[0162] Figure 4 The attribution diagnosis and graded self-healing process shown can be implemented based on a pre-configured training performance attribution and fault self-healing system architecture, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the training performance attribution and fault self-healing system architecture proposed according to the present invention, wherein the system shown includes a signal layer, a diagnosis layer, a processing layer, and a linkage layer: The signal layer is responsible for collecting raw runtime signals such as DataLoader queue level, PCIe / NVLink hardware counters, CUDA kernel and NCCL events, hardware and training metrics; The diagnostic layer includes a step aligner, a wait category decomposition engine, and a multi-source fault classifier, which are respectively responsible for aligning signals according to the training step boundaries, performing wall clock time decomposition of mutually exclusive wait categories, and fusing hardware, communication, and training state signals and combining the proportion vector output by the wait category decomposition engine with the prescription verification results output by the prescription library automatic verification component to determine the fault type. The processing layer includes an optimized prescription database automatic verification component and a hierarchical recovery strategy table online learning component, which are respectively responsible for the attribution result-driven prescription automatic verification closed loop and the recovery cost write-back-driven strategy online learning. The linkage layer is responsible for associating the fault recovery status with the billing policy adjustment of the billing system, and generating an auditable "fault-recovery-cost" record link.

[0163] Based on the above embodiments, the method proposed in this invention can achieve at least the following technical effects: 1. The mean time to recovery (MTTR) of faults is reduced from hours to minutes: Because the detection-location-handling form a fully automated closed loop, and the recovery process can be further compressed by the lazy loading checkpoint mechanism, the mean time to recovery of faults for computing power running tasks is expected to be reduced from hours under the existing manual troubleshooting and handling scheme to minutes (estimated value, the specific value varies with the fault type and checkpoint scale).

[0164] 2. Slow nodes no longer drag down global training throughput: After a slow node is detected, the system dynamically offloads its share of parallel data load to other nodes, replacing the existing "shutdown and card replacement" approach that causes the entire task to pause and wait. Therefore, the computing power task can continue to run at a near-normal throughput before the slow node problem is resolved. The throughput drop caused by slow nodes is expected to narrow from nearly 100% (complete task pause) under the existing shutdown and card replacement scheme to single-digit percentages (estimated value, the specific value varies with the proportion of slow nodes).

[0165] 3. Platform and User Responsibilities Can Be Quantified: The step-level waiting category decomposition provides quantitative evidence of the proportion of various waiting types, including waiting for data, waiting for copying, waiting for computation, waiting for communication, and waiting for slow nodes. This evidence satisfies the constraints of completeness (the sum of all components equals the wall clock duration) and mutual exclusivity. Therefore, this quantitative evidence can directly support the determination of responsibility for deductions in disputes over billing for decreased training efficiency, replacing the existing method of responsibility determination that relies on manual comparison of scattered resource utilization curves and is difficult to provide definitive conclusions. Furthermore, the retention / rollback results of candidate prescriptions provide a mechanism for determining responsibility: prescription retention indicates user-side elimination, while prescription rollback and escalation to a fault indicates platform-side responsibility. Responsibility determination no longer relies on manual interpretation.

[0166] Please see Figure 6 , Figure 6 This is a structural diagram of a self-healing device for abnormal computing power operation tasks on an intelligent computing cloud platform provided by the present invention, as shown in the figure. Figure 6 As shown, the intelligent computing cloud platform's computing power operation task anomaly self-healing device 600 includes: The signal acquisition module 601 is used to acquire multi-dimensional runtime signals and computing resource status signals generated by the model in the intelligent computing cloud platform during the execution of computing power tasks; The first determining module 602 is used to time-align the multidimensional runtime signal according to the boundary of the training step to obtain the wall clock duration of each training step, and decompose the wall clock duration to determine multiple mutually exclusive waiting categories and the duration of the waiting category. The second determining module 603 is used to determine the proportion of time spent by each waiting category in the wall clock duration, and match each proportion of time spent with the triggering conditions of each candidate exception handling strategy to determine the matching target exception handling strategy, and execute the target exception handling strategy to obtain the execution result. The third determining module 604 is used to determine the target fault type based on the computing resource status signal and the execution result, determine the target recovery action corresponding to the target fault type, and execute the target recovery action to perform abnormal self-healing of the computing power operation task.

[0167] In one embodiment, the multidimensional runtime signals include at least a data loading queue level signal, an inter-device data copy hardware counter signal, a computation kernel function execution timeline signal, and a collection communication event log signal. The multidimensional runtime signals are all continuously recorded throughout the entire process of the computing power operation task in the form of interval events carrying timestamps.

[0168] In one embodiment, the first determining module 602 is specifically used for: All interval events recorded by the multidimensional runtime signals within the wall clock duration of the training step are merged and sorted according to the starting timestamp to obtain the event sequence on a unified time axis; Identify state transition points in an event sequence, where a state transition point is the moment when either an execution state or a waiting state ends and another state begins; Based on the state transition points, the time axis of the training step is divided into multiple state intervals that cover the entire training step interval, wherein no two state intervals overlap. Determine the waiting category to which each state interval belongs, and sum the durations of all state intervals under the same waiting category to obtain the time consumption of each waiting category. The waiting categories include at least: waiting for data loading, waiting for data copying, waiting for computation, and waiting for communication.

[0169] In one embodiment, the waiting category further includes waiting for slow nodes, which characterizes the additional waiting caused by nodes with abnormal computation time; the first determining module 602 is further configured to: Obtain the computation time of each computing node participating in the training step; Determine the median computation time for all computing nodes; Calculate the ratio of the computation time of each computing node to the median; When the ratio of the target computing power node is greater than the first preset threshold for more than a first preset number of training steps, the target computing power node is determined to be a slow node. The portion of the computation time of the target computing node that exceeds the median is included in the time spent waiting for the slow node, and the corresponding excess portion is deducted from the time spent waiting for computation.

[0170] In one embodiment, candidate exception handling strategies and corresponding triggering conditions are pre-configured in a strategy library; wherein, the second determining module 603 is used to: Determine the second preset threshold and the second preset quantity corresponding to the different waiting categories configured for each trigger condition from the policy library; When the proportion of time spent in the waiting category is greater than the second preset threshold for more than a second preset number of training steps, the corresponding triggering condition is determined to be met, and the associated candidate exception handling strategy is taken as the target exception handling strategy.

[0171] In one embodiment, the second determining module 603 is further configured to: Determine the change in the proportion of each waiting category before and after applying the target exception handling strategy; If the magnitude of change meets the preset improvement conditions, the configuration corresponding to the target anomaly handling strategy is retained, and the result of retaining the configuration is used as the execution result. If the magnitude of the change does not meet the improvement conditions, the configuration will be rolled back to the state before the target exception handling strategy was applied, and the result of the rollback configuration will be used as the execution result.

[0172] In one embodiment, the computing resource status signal includes a hardware status signal, a communication status signal, and a training status signal; wherein, the third determining module 604 is specifically used for: If the hardware status signals and / or communication status signals meet the preset hard fault criteria, the target fault type is determined based on the hardware status signals and / or communication status signals. If neither the hardware status signal nor the communication status signal meets the hard fault criterion, and the time consumption ratio of the target waiting category in multiple waiting categories continues to be greater than the third preset threshold for more than a third preset number of training steps, and the corresponding target anomaly handling strategy is rolled back, then the target fault type is determined based on the computing resource status signal associated with the target waiting category.

[0173] In one embodiment, the fault types include software anomalies, single-card hardware failures, slow nodes, and link hotspots. A hierarchical recovery strategy table is pre-built to record the correspondence between fault types and candidate recovery actions of different interruption levels, as well as the expected cost estimate for each candidate recovery action. The third determining module 604 is also used for: Based on the graded recovery strategy table, the candidate recovery action with the smallest expected cost estimate is selected as the target recovery action from at least one candidate recovery action corresponding to the target fault type.

[0174] In one embodiment, the candidate recovery action includes at least one of the following: If the fault type is a software exception, then the training process will be restarted in place. If the fault type is a single card hardware fault, then isolate the faulty card, replace it with a replacement card, and perform recovery based on checkpoints; If the fault type is a slow node, the data parallel load share borne by the slow node will be dynamically offloaded to other computing nodes without stopping the computing power to run tasks. If the fault type is a link hotspot, then a communication ring topology reconfiguration is triggered to bypass the abnormal physical link; The interruption of the computing power operation task caused by the dynamic unloading and the communication ring topology reconstruction is lower than that caused by the in-situ restart of the training process. The interruption of the in-situ restart of the training process is lower than that caused by the isolation fault card, the replacement card and the recovery based on checkpoints.

[0175] In one embodiment, the apparatus further includes: The fourth determination module is used to determine the actual cost of performing the target recovery action and update the hierarchical recovery strategy table online based on the actual cost.

[0176] In one embodiment, the fourth determining module is specifically used for: Obtain the actual cost incurred by executing the target recovery action to handle the target fault type. The actual cost is determined based on the time consumed by the target recovery action and the amount of computational loss. The amount of computational loss is determined based on the waiting time of slow nodes and the waiting time of communication for each training step during the period from the occurrence of the fault to the completion of the recovery. Based on the preset learning rate, the expected cost estimate of the target recovery action is weighted and summed with the actual cost to obtain the updated expected cost estimate; The updated expected cost estimate is written back to the tiered recovery strategy table to update the tiered recovery strategy table.

[0177] In one embodiment, the fourth determining module is further configured to: The occupancy status of the computing resources corresponding to the target recovery action is marked as a specific status, so that the billing system can perform billing policy adjustment operations based on the specific status; The responsibility attribution is determined based on the execution results of the target anomaly handling strategy; Generate and record information including target fault type, trigger time, target recovery action, recovery time, actual cost, and responsibility attribution determination result.

[0178] In one embodiment, the fourth determining module is specifically used for: If the execution result is to retain the configuration of the target exception handling strategy, then it is determined to be the user's responsibility; If the execution result is a rollback to the configuration prior to the application's target exception handling strategy and triggers the determination of the target fault type, then it is determined to be the responsibility of the platform side.

[0179] In one embodiment, the apparatus further includes: The fifth determination module is used to determine the change in the proportion of waiting time associated with the target fault type before and after fault recovery after the target recovery action is completed, and to determine the fault recovery effect based on the change in proportion information.

[0180] In this way, by collecting multi-dimensional signals and performing mutual exclusion decomposition on a unified time axis, performance bottlenecks can be accurately quantified. Then, by combining the execution results of the anomaly handling strategy with the computing resource status signals, a complete closed loop from performance attribution to fault self-healing can be achieved. This can accurately identify and automatically repair various anomalies in computing power operation tasks without interrupting the main training process as much as possible, effectively reducing the cost of manual intervention and the loss of idle computing power.

[0181] It should be noted that the self-healing device for abnormal computing power operation tasks of the intelligent computing cloud platform in this invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0182] The present invention also provides an electronic device, see [link to relevant documentation]. Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 701, a processor 702, and a program or instructions stored in the memory 701 that run on the memory. When the program or instructions are executed by the processor 702, they can achieve the following: Figure 1 The steps in the corresponding intelligent computing cloud platform's computing power operation task anomaly self-healing method embodiment and the achievement of the same beneficial effect will not be repeated here.

[0183] The processor 702 can be a CPU, ASIC, FPGA, or GPU.

[0184] Those skilled in the art will understand that all or part of the steps of the above-described embodiment of the self-healing method for abnormal operation of computing power on the intelligent computing cloud platform can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0185] The present invention also provides a readable storage medium on which a computer program is stored, and which, when executed by a processor, can perform the above-described functions. Figure 1 Any step in the embodiment of the self-healing method for abnormal task operation on the corresponding intelligent computing cloud platform can achieve the same technical effect, and will not be described again here to avoid repetition. The storage medium mentioned includes, for example, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0186] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes of the corresponding intelligent computing cloud platform's computing power operation task anomaly self-healing method embodiment can achieve the same technical effect, and will not be described again here to avoid repetition.

[0187] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this invention indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0188] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of the present invention.

[0190] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other modifications under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these modifications are within the protection scope of the present invention.

Claims

1. A method for self-healing anomalies in computing power operation tasks on an intelligent computing cloud platform, characterized in that, include: Step S1: Collect multi-dimensional runtime signals and computing resource status signals generated by the model in the intelligent computing cloud platform during the computing power operation task; Step S2: Align the multidimensional runtime signal according to the boundary of the training step to obtain the wall clock duration of each training step, and decompose the wall clock duration to determine multiple mutually exclusive waiting categories and the duration of the waiting category. Step S3: Determine the time consumption ratio of each waiting category in the wall clock duration, and match each time consumption ratio with the triggering conditions of each candidate exception handling strategy to determine the matching target exception handling strategy, and execute the target exception handling strategy to obtain the execution result; Step S4: Determine the target fault type based on the computing resource status signal and the execution result, determine the target recovery action corresponding to the target fault type, and execute the target recovery action to perform the abnormal self-healing of the computing power operation task.

2. The method as described in claim 1, characterized in that, The multidimensional runtime signals include at least the data loading queue level signal, the hardware counter signal for data copying between devices, the execution timeline signal of the computation kernel function, and the set communication event log signal. The multidimensional runtime signals are continuously recorded throughout the entire process of the computing power operation task in the form of interval events carrying timestamps.

3. The method as described in claim 2, characterized in that, Step S2 includes: Step S2.1: Merge and sort all interval events recorded by the multidimensional runtime signal within the wall clock duration of the training step according to the start timestamp to obtain an event sequence on a unified time axis; Step S2.2: Identify the state transition points in the event sequence, wherein the state transition point is the moment when any execution state or waiting state ends and another state begins; Step S2.3: Based on the state transition points, divide the time axis of the training step into multiple state intervals that cover the entire training step interval, wherein no two state intervals overlap. Step S2.4: Determine the waiting category to which each state interval belongs, and sum the durations of all state intervals under the same waiting category to obtain the time consumption of each waiting category. The waiting category includes at least: waiting for data loading, waiting for data copying, waiting for calculation, and waiting for communication.

4. The method as described in claim 3, characterized in that, The waiting category also includes waiting for slow nodes, which are used to characterize the additional waiting caused by nodes with abnormal computation time; step S2 also includes: Step S2.5: Obtain the computation time of each computing node participating in the training step; Step S2.6: Determine the median computation time of all the computing nodes; Step S2.7: Calculate the ratio of the computation time of each computing node to the median; Step S2.8: When the ratio of the target computing power node is continuously greater than the first preset threshold for more than a first preset number of training steps, the target computing power node is determined to be a slow node. Step S2.9: Include the portion of the computation time of the target computing node that exceeds the median into the time spent waiting for the slow node, and deduct the corresponding excess portion from the time spent waiting for computation.

5. The method as described in claim 1, characterized in that, The candidate exception handling strategies and the triggering conditions corresponding to each candidate exception handling strategy are pre-configured in the strategy library; wherein, step S3 includes: Step S3.1: Determine the second preset threshold and the second preset quantity corresponding to the different waiting categories of each trigger condition configuration from the strategy library; Step S3.2: When the time consumption ratio of the waiting category is greater than the second preset threshold for more than the second preset number of training steps in a row, it is determined that the corresponding triggering condition is met, and the associated candidate exception handling strategy is taken as the target exception handling strategy.

6. The method as described in claim 1, characterized in that, Step S3 further includes: Step S3.3: Determine the change in the proportion of each waiting category before and after applying the target anomaly handling strategy; Step S3.4: If the magnitude of the change meets the preset improvement conditions, then retain the configuration corresponding to the target anomaly handling strategy, and use the result of retaining the configuration as the execution result; Step S3.5: If the change magnitude does not meet the improvement conditions, roll back to the configuration before the target anomaly handling strategy was applied, and use the result of the rollback configuration as the execution result.

7. The method as described in claim 1, characterized in that, The computing resource status signals include hardware status signals, communication status signals, and training status signals; wherein, step S4 includes: Step S4.1: If the hardware status signal and / or the communication status signal meet the preset hard fault criteria, then determine the target fault type based on the hardware status signal and / or the communication status signal. Step S4.2: If neither the hardware status signal nor the communication status signal satisfies the hard fault criterion, and the time consumption ratio of the target waiting category among the multiple waiting categories continues to be greater than the third preset threshold for more than a third preset number of training steps, and the corresponding target anomaly handling strategy is rolled back, then the target fault type is determined based on the computing resource status signal associated with the target waiting category.

8. The method as described in claim 1, characterized in that, The fault types include software anomalies, single-card hardware failures, slow nodes, and link hotspots. A hierarchical recovery strategy table is pre-built to record the correspondence between the fault types and candidate recovery actions of different interruption levels, as well as the expected cost estimate for each candidate recovery action. Step S4 further includes: Step S4.3: According to the hierarchical recovery strategy table, determine the candidate recovery action with the smallest expected cost estimate from at least one candidate recovery action corresponding to the target fault type as the target recovery action.

9. The method as described in claim 8, characterized in that, The candidate recovery action includes at least one of the following: If the fault type is a software-related exception, then the training process will be restarted in place. If the fault type is a single-card hardware fault, then isolate the faulty card, replace the card, and perform recovery based on checkpoints; If the fault type is the slow node, then without stopping the computing power operation task, the data parallel load share borne by the slow node will be dynamically offloaded to other computing power nodes. If the fault type is the link hotspot, then a communication ring topology reconfiguration is triggered to bypass the abnormal physical link; The interruption of the computing power operation task caused by the dynamic unloading and the communication ring topology reconstruction is lower than that caused by the in-situ restart of the training process. The interruption of the in-situ restart of the training process is lower than that caused by the isolation fault card, the replacement card and the recovery based on checkpoints.

10. The method as described in claim 8, characterized in that, The method further includes: Step S5: Determine the actual cost of performing the target recovery action, and update the graded recovery strategy table online based on the actual cost.

11. The method as described in claim 10, characterized in that, Step S5 includes: Step S5.1: Obtain the actual cost generated by performing the target recovery action to process the target fault type, wherein the actual cost is determined based on the time consumed by the target recovery action and the amount of computational loss, and the amount of computational loss is determined based on the waiting time of slow nodes and the waiting time of communication for each training step during the period from the occurrence of the fault to the completion of recovery. Step S5.2: Based on a preset learning rate, the expected cost estimate of the target recovery action is weighted and summed with the actual cost to obtain an updated expected cost estimate; Step S5.3: Write the updated expected cost estimate back to the tiered recovery strategy table to update the tiered recovery strategy table.

12. The method as described in claim 11, characterized in that, Step S5 further includes: Step S5.4: Mark the occupancy status of the computing resources corresponding to the execution of the target recovery action as a specific status, so that the billing system can perform billing policy adjustment operations based on the specific status; Step S5.5: Determine the responsibility attribution based on the execution result of the target anomaly handling strategy; Step S5.6: Generate record information that records the target fault type, trigger time, target recovery action, recovery time, actual cost, and responsibility attribution determination result.

13. The method as described in claim 12, characterized in that, Step S5.5 includes: Step S5.5.1: If the execution result is to retain the configuration of the target exception handling strategy, then it is determined to be the user-side responsibility; Step S5.5.2: If the execution result is to roll back to the configuration before the application of the target exception handling strategy and trigger the determination of the target fault type, then it is determined to be the responsibility of the platform side.

14. The method as described in claim 1, characterized in that, The method further includes: Step S6: After the target recovery action is completed, determine the change in the proportion of the waiting category associated with the target fault type before and after the fault recovery, and determine the fault recovery effect based on the change in the proportion information.

15. A self-healing device for abnormal computing power operation tasks in an intelligent computing cloud platform, characterized in that, include: The signal acquisition module is used to acquire multi-dimensional runtime signals and computing resource status signals generated by the model in the intelligent computing cloud platform during the execution of computing power tasks; The first determining module is used to time-align the multidimensional runtime signal according to the boundary of the training step to obtain the wall clock duration of each training step, and decompose the wall clock duration to determine multiple mutually exclusive waiting categories and the duration of the waiting category. The second determining module is used to determine the proportion of time spent by each waiting category in the wall clock duration, and match each proportion of time spent with the triggering conditions of each candidate exception handling strategy to determine the matching target exception handling strategy, and execute the target exception handling strategy to obtain the execution result. The third determining module is used to determine the target fault type based on the computing resource status signal and the execution result, determine the target recovery action corresponding to the target fault type, and execute the target recovery action to perform the abnormal self-healing of the computing power operation task.

16. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the self-healing method for abnormal computing power operation tasks of the intelligent computing cloud platform as described in any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the self-healing method for abnormal computing power operation tasks of the intelligent computing cloud platform as described in any one of claims 1 to 14.

18. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps of the self-healing method for abnormal operation of computing power tasks on an intelligent computing cloud platform as described in any one of claims 1 to 14.

Images