A cloud model task state identification type exception handling method

Abstract

The application discloses a cloud model task state recognition type abnormality processing method, and belongs to the technical field of cloud task management. Process collection is performed on cloud model task operation data, and a task state evidence chain is formed. An evidence front is extracted based on a preset task state template, and task state recognition is completed in combination with a limited detection action. An abnormal starting point is located, state consistency is checked back, and differentiated abnormality treatment is performed after a consistent recovery boundary is determined. The method can distinguish between a normal long-time state and an abnormal stagnation, reduces the risk of false restart and progress loss, improves the operation stability and abnormal recovery reliability of large model training tasks, and is suitable for cloud multi-node large model training, fine tuning and other scenes.

Description

Technical Field

[0001] This invention relates to the field of cloud task management technology, and in particular to a cloud-based model task status identification-based anomaly handling method, electronic device, and storage medium. Background Technology

[0002] As large model training and fine-tuning tasks are deployed to the cloud, elastic GPU cloud clusters are widely used to perform multi-node collaborative training tasks. These tasks may be affected by factors such as node changes, GPU failures, network fluctuations, and object storage latency.

[0003] Existing technologies typically determine whether cloud-based model tasks are abnormal by means of heartbeat detection, training step count monitoring, GPU utilization monitoring, network traffic monitoring, log update time monitoring, and checkpoint save time monitoring. When the monitored indicators exceed preset thresholds, they will perform container restart, workgroup restart, node replacement, or resume training from the most recent checkpoint.

[0004] However, the main problem with existing technologies in practical applications is the difficulty in accurately distinguishing between normal, prolonged periods of stagnation and abnormal stalling states in cloud-based model tasks. During fine-tuning of large multi-node models, long-duration computations, gradient synchronization, checkpoint uploads, inventory submissions, data source waiting, and model resharding can all manifest as a temporary halt in training steps; similarly, communication blockages, node disconnections, GPU anomalies, or incomplete checkpoints can produce the same external behavior. Existing technologies that rely solely on heartbeats, resource utilization, or fixed timeouts to determine anomalies can easily lead to the accidental restart of normal tasks or the failure to detect latent stalls in a timely manner. Furthermore, directly loading the most recent checkpoint or performing a complete rollback after an anomaly without verifying the consistency of model parameters, optimizer status, random number status, data read locations, and checkpoint sharding can easily result in incomplete recovery, duplicate data processing, or loss of completed training progress. Therefore, this invention proposes a cloud-based model task state-based anomaly handling method. Summary of the Invention

[0005] The purpose of this invention is to provide a cloud-based model task state recognition-based anomaly handling method to solve the above-mentioned technical problems.

[0006] In a first aspect, to solve the above-mentioned technical problems, the present invention adopts the following technical solution: a cloud-based model task state recognition-based anomaly handling method, comprising:

[0007] Obtain the runtime data of the cloud-based model tasks and aggregate the task process to obtain the basic data of the model tasks;

[0008] The basic data of the model task are organized into an evidence chain to obtain the task status evidence chain;

[0009] Based on the preset task status template, the evidence frontier of the task status evidence chain is extracted to obtain the current evidence frontier;

[0010] Based on the current evidence frontier, state identification is performed to obtain the task state identification result;

[0011] Based on the task state identification result, the task state evidence chain is restored to determine the boundary, and a consistent restoration boundary result is obtained.

[0012] Anomaly handling is performed based on the task state identification results and consistent recovery boundary results to obtain the model task anomaly handling results.

[0013] As a preferred implementation, the basic data of the model task is organized into an evidence chain to obtain a task state evidence chain, including:

[0014] The basic data of the model task are parsed to obtain a set of task process events;

[0015] Based on the preset task stages, the set of task process events is state-mapped to obtain a set of state events.

[0016] The set of state events is processed into evidence units to obtain task state evidence units;

[0017] Based on the execution dependencies between the task status evidence units, association tags are performed to obtain status evidence association results;

[0018] Based on the association results of the state evidence, a chain is formed to obtain the task state evidence chain.

[0019] As a preferred implementation, association tagging is performed based on the execution dependencies between the task state evidence units, including:

[0020] Node attribution identification is performed on the task status evidence unit to obtain the node evidence attribution result;

[0021] Based on the node evidence attribution results, the task status evidence units within the same working node are sequentially matched to obtain the intra-node dependency relationship.

[0022] Based on the task phase transition relationship, the task status evidence units between different working nodes are collaboratively matched to obtain the inter-node dependency relationship.

[0023] The consistency of the intra-node dependencies and inter-node dependencies is verified to obtain the valid dependencies.

[0024] The valid dependencies are associated and marked to obtain the state evidence association results.

[0025] As a preferred implementation, consistency checks are performed on the intra-node dependencies and inter-node dependencies to obtain valid dependencies, including:

[0026] The training progress continuity is verified for the dependencies within the nodes to obtain the verification results within the nodes.

[0027] Perform a coordination phase consistency check on the dependencies between the nodes to obtain the node check results;

[0028] Based on the intra-node verification results and inter-node verification results, task process consistency matching is performed to obtain dependency matching results.

[0029] The execution order validity is filtered based on the dependency matching results to obtain valid dependencies.

[0030] As a preferred implementation, the task state evidence chain is subjected to evidence front extraction based on a preset task state template to obtain the current evidence front, including:

[0031] The preset task state template is parsed to obtain a set of template evidence items;

[0032] Based on the set of template evidence items, the task status evidence chain is matched to obtain the evidence matching result;

[0033] The evidence matching results are marked with evidence arrival information to obtain evidence arrival results;

[0034] Based on the arrival results of the aforementioned evidence, the latest valid evidence for each working node is screened to obtain the node evidence frontier;

[0035] The current evidence front is obtained by aligning the node evidence front with the task phase.

[0036] As a preferred implementation, the latest valid evidence for each working node is screened based on the evidence arrival results, including:

[0037] The results of the evidence arrival are categorized into working nodes to obtain a node evidence set;

[0038] Arrival status identification is performed on the node evidence set to obtain the evidence of arrived nodes;

[0039] The validity of the arrived node evidence is determined to obtain valid node evidence.

[0040] Based on the valid node evidence, the state advancement position is filtered to obtain the latest valid evidence;

[0041] The latest valid evidence corresponding to each working node is organized into a frontier to obtain the node evidence frontier.

[0042] As a preferred implementation, state identification is performed based on the current evidence frontier to obtain task state identification results, including:

[0043] The current evidence frontier is analyzed using state evidence analysis to obtain frontier evidence features;

[0044] Candidate task states are obtained by matching the frontier evidence features with candidate states based on a preset task state template.

[0045] The candidate task states are distinguished to obtain state distinction results;

[0046] Based on the state differentiation results, a difference evidence analysis is performed on the candidate task states to obtain restricted detection actions;

[0047] Based on the detection response results of the restricted detection action, the candidate task states are matched accordingly to obtain the task state identification results.

[0048] As a preferred implementation, based on the state differentiation results, differential evidence analysis is performed on the candidate task states to obtain restricted detection actions, including:

[0049] The state distinction results are used to filter candidate states to obtain candidate states to be distinguished;

[0050] Based on the preset task state template, expected evidence is extracted from the candidate states to be distinguished to obtain candidate state evidence items.

[0051] The candidate state evidence items are compared for evidence differences to obtain the difference evidence items;

[0052] Based on the difference evidence items, the detection type is matched to obtain candidate detection actions;

[0053] The candidate detection actions are subjected to training state influence constraints to obtain restricted detection actions.

[0054] As a preferred implementation, the task state evidence chain is restored based on the task state identification result to obtain a consistent restoration boundary result, including:

[0055] Based on the task status identification result, the abnormal starting point of the task status evidence chain is located to obtain abnormal starting point evidence.

[0056] Based on the abnormal starting point evidence, the candidate aspect backtracking is performed on the task state evidence chain to obtain a set of candidate recovery aspects.

[0057] The training state consistency check is performed on the candidate recovery section set to obtain the state consistency check result.

[0058] Based on the state consistency verification results, the set of candidate recovery aspects is filtered for recovery availability to obtain a set of available recovery aspects.

[0059] The recovery progress of the available recovery sections is optimized to obtain the target recovery section;

[0060] Based on the target restoration section, the restoration boundary organization is performed to obtain a consistent restoration boundary result.

[0061] As a preferred implementation, the task state evidence chain is backtracked based on the abnormal starting point evidence to obtain a candidate recovery aspect set, including:

[0062] The link position of the abnormal starting point evidence in the task status evidence chain is located to obtain the starting point link position;

[0063] Based on the starting link position, reverse dependency tracing is performed on the task status evidence chain to obtain the backtracking evidence path;

[0064] The backtracking evidence path is filtered to obtain candidate recovery nodes;

[0065] Based on the candidate recovery nodes, the sectional range is divided to obtain the candidate recovery section;

[0066] The candidate recovery sections are organized into a set to obtain a set of candidate recovery sections.

[0067] Secondly, this application provides an electronic device, comprising:

[0068] A processor; and a memory storing computer instructions that, when executed by the processor, cause the electronic device to perform the method disclosed in the first aspect above.

[0069] Thirdly, this application provides a computer-readable storage medium containing program instructions that, when executed by a processor, cause the method disclosed in the first aspect to be implemented.

[0070] Compared to existing technologies, this invention, through the collection of task execution data and the organization of state evidence chains, records the entire chain of states, including model computation, data reading, node communication, and checkpoint processing, in association. Combined with evidence frontier extraction and multi-dimensional state matching, it avoids relying solely on heartbeats, resource utilization, or training steps to determine task state, thus improving the accuracy of state identification for cloud-based model tasks. A restricted detection mechanism performs secondary verification of ambiguous states, distinguishing between normal states such as normal long-duration computation, gradient synchronization, and checkpoint uploading, and fault states such as communication blockage, node disconnection, and checkpoint anomalies. This reduces the probability of erroneous task restarts, enhances the ability to detect hidden stagnation, and reduces ineffective waste of computing resources. By backtracking from the anomaly starting point and performing multi-dimensional consistency checks to determine the consistency recovery boundary, anomaly recovery no longer solely relies on the nearest checkpoint, reducing issues such as incomplete checkpoint loading, duplicate data processing, and lost training progress. This improves the reliability of anomaly recovery and the continuity of task execution, while reducing the time cost of operation and maintenance troubleshooting and recovery. Attached Figure Description

[0071] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation on the scope of this application.

[0072] Figure 1 This is a flowchart of a cloud-based model task state recognition-based exception handling method provided in an embodiment of the present invention;

[0073] Figure 2 This is a flowchart illustrating the task status evidence chain organization provided in an embodiment of the present invention.

[0074] Figure 3 A flowchart for evidence frontier extraction and task state identification provided in this embodiment of the invention;

[0075] Figure 4 A flowchart for determining the consistent recovery boundary and handling anomalies provided in this embodiment of the invention. Detailed Implementation

[0076] To make the technical means, creative features, and achieved objectives and effects of this invention easier to understand, the invention is further described below with reference to specific embodiments. However, the following embodiments are merely preferred embodiments of this invention and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments described herein without creative effort are all within the protection scope of this invention. Unless otherwise specified, the experimental methods in the following embodiments are conventional methods, and the materials and reagents used in the following embodiments are commercially available unless otherwise specified.

[0077] like Figure 1As shown in the figure, this embodiment provides a cloud-based model task status recognition-based anomaly handling method.

[0078] In this embodiment, the method includes the following steps:

[0079] S1. Obtain the running data of the cloud model task and collect the task process to obtain the basic data of the model task.

[0080] In this embodiment, the operation data of the cloud model task is obtained, including task configuration data, worker node status data, training progress data, data reading data, communication process data, checkpoint process data, and resource operation data.

[0081] Task ID identification is performed on the running data to obtain the task data set corresponding to the same cloud model task; node identifier extraction is performed on the task data set to obtain the node data set corresponding to each working node; task stage marking is performed on the node data set to obtain stage running data; task process identifier extraction is performed on the stage running data to obtain process identifier data containing training steps, communication sequence number, data reading cursor and checkpoint generation; the stage running data and process identifier data are structured and organized to obtain the basic data of the model task.

[0082] The task configuration data includes task number, model version, training framework, number of worker nodes, parallel group configuration, and checkpoint retention period; the training progress data includes training steps, micro-batch number, optimizer update sequence number, data read cursor, and random number status summary; the communication process data includes communication group number, communication type, communication start sequence number, communication completion sequence number, and current communication sequence number of each worker node; and the checkpoint process data includes checkpoint generation, shard number, shard write event, shard summary, object storage confirmation event, and checkpoint list commit event.

[0083] S2, such as Figure 2 As shown, the basic data of the model task are organized into an evidence chain to obtain the task state evidence chain.

[0084] In this embodiment, the evidence chain is organized for the basic data of the model task, including:

[0085] S21. Perform event parsing on the basic data of the model task to obtain a set of task process events.

[0086] Specifically, the basic data of the model task is split into event fields to obtain event source field, event type field, node identifier field, task stage field, task progress field, and event content field; the event type field is identified to obtain data reading event, model calculation event, communication synchronization event, optimizer update event, checkpoint processing event, node change event, and result submission event; the task progress field is processed by extracting process number to obtain training steps, communication sequence number, data reading cursor, and checkpoint generation; the event type, node identifier, task stage, and task progress number are organized to obtain the task process event set.

[0087] S22. Based on the preset task stage, perform state mapping on the set of task process events to obtain a set of state events.

[0088] Specifically, the task process event set is matched with the stage type, mapping data reading events to the data waiting stage, model calculation events to the forward calculation stage or backward calculation stage, communication synchronization events to the gradient synchronization stage, optimizer update events to the optimizer update stage, checkpoint processing events to the checkpoint sharding upload stage or checkpoint list submission stage, node change events to the model re-sharding stage, and result submission events to the result submission stage, thus obtaining the state event set.

[0089] S23. Perform evidence unitization processing on the set of state events to obtain task state evidence units.

[0090] By constructing evidence fields for each state event in the state event set, an evidence structure is generated that includes task number, work node number, task stage identifier, event type identifier, local consecutive sequence number, previous event summary, training steps, communication sequence number, data read cursor, checkpoint generation, event time, and event content summary. The evidence structure is then subjected to field integrity verification, and invalid events for which the task number or node number cannot be determined are removed. The evidence structures that pass the field integrity verification are uniformly encoded to obtain the task state evidence unit.

[0091] S24. Based on the execution dependency relationship between the task status evidence units, perform association marking to obtain the status evidence association result.

[0092] Further, S241, the node attribution identification is performed on the task status evidence unit to obtain the node evidence attribution result;

[0093] The node attribution identification for task status evidence units includes: extracting the working node number, parallel group number, and node change record from the task status evidence unit; determining the working node to which the evidence unit belongs based on the working node number; determining the communication group or model shard group to which the evidence unit belongs based on the parallel group number; and when there is a node replacement record, correcting the node attribution of the evidence unit based on the node exit time, node joining time, and model shard succession relationship to obtain the node evidence attribution result.

[0094] S242. Based on the node evidence attribution result, sequential matching is performed on the task status evidence units within the same working node to obtain intra-node dependencies. Specifically, sequential matching of task status evidence units within the same working node includes: sorting task status evidence units within the same working node according to local consecutive sequence numbers; performing pre-event summary matching on adjacent sorted task status evidence units; when the pre-event summary of the latter task status evidence unit points to the former task status evidence unit, and the training steps, data reading cursors, or checkpoint generations of both meet the task process progression threshold requirements, an intra-node dependency is established. The engineering definition of the task process progression threshold is the minimum progression amount required to determine whether the task process between adjacent task status evidence units is progressing normally. In this embodiment, the typical value is a progression of 1 step in training steps, 1 communication sequence number, 1 micro-batch data volume in data reading cursors, or 1 generation in checkpoint generations. This threshold is calibrated offline based on the event generation rules of the training framework and the task process continuity requirements. In practical applications, it can be adjusted from 1 to 3 steps depending on the training framework version and parallel strategy. Adjust within the step range; when there is a break in local consecutive sequence numbers, a regression in training steps, or a missing summary of preceding events, mark the corresponding dependencies as abnormal.

[0095] S243. Based on the task phase transition relationship, perform collaborative matching of the task status evidence units between different working nodes to obtain the inter-node dependency relationship. This includes: matching the reverse calculation completion evidence and communication start evidence of each working node based on the communication group number; matching the communication start evidence and communication completion evidence of each working node based on the communication sequence number; matching the fragment completion evidence and checkpoint list generation evidence of each working node based on the checkpoint generation; and matching the fragment migration evidence of the original working node and the fragment reception evidence of the new working node based on the model fragment inheritance relationship to obtain the inter-node dependency relationship.

[0096] S244. Perform consistency checks on the intra-node dependencies and inter-node dependencies to obtain valid dependencies.

[0097] include:

[0098] S2441. Perform training progress continuity verification on the intra-node dependencies to obtain intra-node verification results; specifically, by performing training progress continuity verification on the intra-node dependencies, determine whether the training steps, micro-batch number, optimizer update sequence number, data read cursor, and local continuous sequence number in adjacent task state evidence units satisfy the progressive relationship to obtain intra-node verification results.

[0099] S2442. Perform a coordination phase consistency check on the inter-node dependency relationship, determine whether each working node in the same communication group enters the same communication sequence number, whether the fragment number and fragment digest of each working node under the same checkpoint generation correspond, and whether the fragment migration evidence and fragment reception evidence in the model re-fragmentation process match, and obtain the inter-node check result.

[0100] S2443. Based on the intra-node verification results and inter-node verification results, perform task process consistency matching to determine whether the training steps, communication sequence number, data reading cursor and checkpoint generation in the same dependency relationship can form a connection relationship under the same task process, and obtain the dependency matching result.

[0101] S2444. Perform execution order validity screening on the dependency matching results. Retain the dependencies that match and meet the task stage transition order as valid dependencies, and remove the dependencies with reversed execution order, missing preceding evidence, or mismatched task processes to obtain valid dependencies.

[0102] S245. The effective dependencies are associated and marked to obtain the state evidence association result.

[0103] Specifically, association identifiers are set for the preceding task status evidence units and subsequent task status evidence units in the effective dependency relationship; the association identifiers include intra-node sequential association identifiers, inter-node collaborative association identifiers, communication synchronization association identifiers, checkpoint fragmentation association identifiers, and recovery dependency association identifiers; the association identifiers are written into the corresponding task status evidence units to obtain the status evidence association results.

[0104] S25. Based on the association results of the state evidence, a chain is formed to obtain the task state evidence chain.

[0105] Furthermore, based on the state evidence association results, task state evidence units with intra-node sequential association identifiers within the same working node are organized into intra-node evidence chains; task state evidence units with inter-node collaborative association identifiers between different working nodes are organized into inter-node collaborative evidence chains; task state evidence units with checkpoint fragmentation association identifiers under the same checkpoint generation are organized into checkpoint evidence chains; the intra-node evidence chains, inter-node collaborative evidence chains, and checkpoint evidence chains are merged according to the task progress identifier to obtain the task state evidence chain.

[0106] The task state evidence chain is generated based on model calculation events, data reading events, cross-node communication events, checkpoint processing events, and worker node state data. Specifically, through event parsing, state mapping, evidence unitization, execution dependency association marking, and chain-like organization, scattered runtime events are transformed into a state evidence structure with sequential dependencies. The task state evidence chain includes task state evidence units, intra-node dependencies, inter-node dependencies, communication synchronization associations, checkpoint fragmentation associations, and recovery dependency associations. This task state evidence chain provides traceable and comparable state evidence for subsequent evidence front extraction. Subsequent steps, including evidence front extraction, state identification, and recovery boundary determination, all utilize the evidence units and their dependencies within the task state evidence chain.

[0107] S3, such as Figure 3 As shown, the evidence frontier is extracted from the task status evidence chain according to the preset task status template to obtain the current evidence frontier.

[0108] In this embodiment, the evidence front is used to represent the latest valid task state that each working node has reached in the current task process.

[0109] S31. Perform state evidence parsing on the preset task state template to obtain a set of template evidence items.

[0110] Specifically, the task stages in the preset task state template are split into stages to obtain the data waiting stage, forward computation stage, backward computation stage, gradient synchronization stage, optimizer update stage, checkpoint sharding upload stage, checkpoint list submission stage, model re-sharding stage, and result submission stage; expected evidence is extracted for each task stage to obtain the necessary evidence items, optional evidence items, and prohibited evidence items that should appear in the corresponding stage; and each task stage and its corresponding evidence items are associated and organized to obtain a template evidence item set.

[0111] S32. Based on the template evidence item set, perform evidence matching on the task status evidence chain to obtain the evidence matching result.

[0112] The task status evidence units in the task status evidence chain are read for stage identifiers to obtain evidence stage identifiers; the task status evidence units are read for event types to obtain evidence type identifiers; the evidence stage identifiers and evidence type identifiers are matched with the template evidence item set; when the task status evidence unit meets the stage requirements, type requirements, and task progress requirements of the corresponding template evidence item, a matching success mark is generated; when the evidence required by the template evidence item does not appear in the task status evidence chain, an evidence missing mark is generated, and the evidence matching result is obtained.

[0113] S33. Mark the evidence arrival status of the evidence matching results to obtain the evidence arrival results.

[0114] By identifying the arrival status of the evidence matching results, evidence items that match successfully and satisfy the current task progress are marked as arrived evidence; evidence items that should appear in the current task stage but have not appeared are marked as missing evidence; evidence items that have not appeared even after exceeding the duration of the corresponding task stage are marked as delayed evidence; and evidence items that do not match the current task stage but have already appeared are marked as abnormal evidence. The arrived evidence, missing evidence, delayed evidence, and abnormal evidence are then structured and organized to obtain the evidence arrival results.

[0115] S34. Based on the arrival results of the evidence, the latest valid evidence of each working node is screened to obtain the node evidence frontier.

[0116] S341. Classify the evidence arrival results into working nodes to obtain a node evidence set;

[0117] Read the working node number, parallel group number, and task process identifier corresponding to each evidence item in the evidence arrival result; group the arrived evidence, missing evidence, delayed evidence, and abnormal evidence belonging to the same working node into the same node group according to the working node number; subclassify the evidence involving different communication groups or model fragment groups under the same working node according to the parallel group number; and organize the classified node evidence into a set to obtain the node evidence set.

[0118] S342. Perform arrival status identification on the node evidence set to obtain the evidence of the arrived nodes;

[0119] The arrival markers of each evidence item in the node evidence set are identified, and evidence items marked as arrived are selected. The task progress of the selected evidence items is compared to determine whether their training steps, communication sequence number, data reading cursor or checkpoint generation correspond to the current node task progress. Evidence items with valid arrival markers and corresponding task progress are determined as evidence of the arrived node.

[0120] S343. Determine the validity of the arrived node evidence, including: determining whether the arrived node evidence has a valid event source, valid node affiliation, valid task stage, and valid execution dependency; if the arrived node evidence originates from a training framework event, object storage confirmation event, node control event, or a verified valid resource status event, and has a valid dependency in the task status evidence chain, then it is determined to be valid node evidence.

[0121] S344. Based on the valid node evidence, perform state advancement position filtering, including: reading the task stage identifier and task progress identifier in the valid node evidence; comparing the advancement positions of multiple valid node evidences within the same working node according to the preset task stage order; when multiple valid node evidences belong to the same task stage, determining the evidence with the highest advancement position based on the training steps, communication sequence number, data reading cursor and checkpoint generation; and determining the valid node evidence with the highest advancement position as the latest valid evidence.

[0122] S345. Organize the latest valid evidence corresponding to each working node to obtain the node evidence frontier; arrange the latest valid evidence corresponding to each working node according to the working node number, and organize it according to the parallel group number, task stage identifier, and task progress identifier; uniformly record the latest task stage, latest training steps, latest communication sequence number, latest data reading cursor, and latest checkpoint generation of each working node; use the uniformly recorded latest valid evidence as the current state advancement position of each working node to organize and obtain the node evidence frontier.

[0123] S35. Perform task phase alignment on the node evidence front to obtain the current evidence front.

[0124] Specifically, a stage consistency judgment is performed on the node evidence frontier of each working node to obtain a stage alignment result; when the latest valid evidence of each working node belongs to the same task stage or the same communication round, a consistent state frontier is generated; when some working nodes have entered the subsequent task stage while some working nodes remain in the previous task stage, a bifurcation state frontier is generated; when the training steps have not progressed but the communication evidence, checkpoint evidence, or resharding evidence is still being updated, a continuously updated evidence frontier is generated; the consistent state frontier, bifurcation state frontier, or continuously updated evidence frontier is used as the current evidence frontier.

[0125] The current evidence frontier is generated based on the task state evidence chain and a preset task state template. Specifically, through template evidence item parsing, evidence matching, evidence arrival marking, latest valid evidence screening, and task stage alignment processing, the latest task state reached by each working node is determined. The current evidence frontier includes the latest valid evidence, arrived evidence, missing evidence, delayed evidence, abnormal evidence, and task stage alignment results between nodes for each working node. The current evidence frontier provides a direct basis for distinguishing between normal long-term states and abnormal stagnant states. Subsequent steps extract frontier evidence features based on the current evidence frontier and further generate candidate task states.

[0126] S4. Based on the current evidence frontier, perform state identification to obtain the task state identification result.

[0127] In this embodiment, state recognition includes candidate state matching, state discrimination determination, difference evidence analysis, restricted detection action generation, and detection response matching.

[0128] S41. Perform state evidence analysis on the current evidence frontier to obtain frontier evidence features.

[0129] Specifically, the current evidence frontier is statistically analyzed to obtain characteristics of arrived evidence, missing evidence, delayed evidence, and anomalous evidence; node advancement difference analysis is performed on the current evidence frontier to obtain node advancement difference characteristics; process difference analysis is performed on communication sequence numbers, data reading cursors, and checkpoint generations to obtain task process difference characteristics; runtime status is extracted from GPU utilization, network transmission volume, object storage request status, and node heartbeats to obtain resource response characteristics; and the above characteristics are combined to obtain the frontier evidence characteristics.

[0130] S42. Based on the preset task state template, perform candidate state matching on the frontier evidence features to obtain candidate task states.

[0131] The frontier evidence features are matched with the various task status conditions in the preset task status template. These task status conditions include normal long-term computation conditions, normal gradient synchronization conditions, normal checkpoint upload conditions, normal checkpoint list submission conditions, normal model re-sharding conditions, data source waiting conditions, GPU computation anomaly conditions, ensemble communication stagnation conditions, worker node disconnection conditions, checkpoint sharding anomaly conditions, and checkpoint list anomaly conditions.

[0132] Furthermore, candidate state matching values ​​are calculated based on the coverage of reached evidence, the proportion of missing evidence, the degree of inter-node dependency consistency, and the degree of resource response matching. Task states whose candidate state matching values ​​reach a preset candidate state matching threshold are determined as candidate task states. The engineering definition of the preset candidate state matching threshold is the lower limit of the matching degree for determining whether a task state should be included in the candidate set. In this embodiment, the typical value is 0.60. This threshold is calibrated offline based on the balance between recall and precision of state recognition. In practical applications, it can be adjusted within the range of 0.50 to 0.75 according to the task type and state complexity.

[0133] S43. The candidate task states are distinguished to obtain the state distinction results.

[0134] The difference in matching values ​​between the first candidate task state and the second candidate task state is calculated to obtain the state discrimination score. When the state discrimination score reaches a preset state discrimination threshold, a state-determinable result is generated. When the state discrimination score does not reach the preset state discrimination threshold, a state-ambiguous result is generated to obtain the state-discrimination result.

[0135] S44. Based on the state differentiation results, perform difference evidence analysis on the candidate task states to obtain restricted detection actions.

[0136] S441. The state differentiation results are used to filter candidate states to obtain candidate states to be differentiated; the matching value or differentiation marker corresponding to each candidate task state in the state differentiation results is read; candidate task states whose matching values ​​reach the preset candidate state filtering range and whose differences between them do not reach the preset state differentiation threshold are determined as candidate states to be differentiated; the preset candidate state filtering range is defined as the lower limit range of matching values ​​for filtering candidate states to be differentiated. In this embodiment, the typical value is 0.55 to 0.90. This range is calibrated offline based on the balance between candidate state recall and differentiation efficiency. In practical applications, it can be adjusted within the range of 0.50 to 0.95 according to the number of states and the differentiation difficulty; candidate task states whose matching values ​​are lower than the lower limit of the preset candidate state filtering range or have been excluded by the current evidence frontier are not included in the candidate states to be differentiated, thus obtaining the candidate states to be differentiated.

[0137] S442. Based on the preset task state template, perform expected evidence extraction on the candidate states to be distinguished to obtain candidate state evidence items; specifically, search for the corresponding state template record in the preset task state template according to the candidate states to be distinguished; extract the state evidence that should appear in the candidate state, the state evidence that can be temporarily missing, and the state evidence that should not appear from the state template record; organize the extracted state evidence according to the evidence type to obtain candidate state evidence items, wherein the candidate state evidence items include at least one of the following: calculation response evidence items, communication response evidence items, data reading evidence items, checkpoint processing evidence items, and work node response evidence items.

[0138] S443. Perform evidence difference comparison on the candidate state evidence items, including: comparing evidence items that should appear, allowed missing evidence items, and should not appear under different candidate states to be distinguished; and determining the evidence items that can distinguish different candidate states to be distinguished as difference evidence items. For example, when the candidate states to be distinguished include normal long-term computing state and GPU computing abnormal state, GPU control response evidence is determined as difference evidence items; when the candidate states to be distinguished include normal gradient synchronization state and ensemble communication stagnation state, training communication sequence number advancement evidence and control communication response evidence are determined as difference evidence items; when the candidate states to be distinguished include checkpoint fragmentation abnormal state and checkpoint list abnormal state, fragmentation integrity evidence and list submission evidence are determined as difference evidence items.

[0139] S444. Perform detection type matching based on the difference evidence items to obtain candidate detection actions;

[0140] Specifically, when the difference evidence item is GPU control response evidence, a GPU response probe action is generated; when the difference evidence item is control communication response evidence, a communication control group probe action is generated; when the difference evidence item is data source response evidence, a data source metadata probe action is generated; when the difference evidence item is checkpoint storage response evidence, a checkpoint storage probe action is generated; and when the difference evidence item is worker node control response evidence, a worker node control response probe action is generated.

[0141] S445. Apply training state influence constraints to the candidate detection actions to obtain restricted detection actions;

[0142] Specifically, this includes: determining whether candidate probe actions change model parameters, optimizer state, gradient values, data read cursor, random number state, formal checkpoint file, and formal training output; retaining candidate probe actions that do not change the above training state as restricted probe actions; and eliminating candidate probe actions that may change the above training state.

[0143] The restricted probing actions include at least one of the following: GPU control response query, worker node control plane heartbeat query, communication control group connectivity query, data source metadata read query, object storage checkpoint metadata query, and checkpoint shard summary read query. These restricted probing actions only read control responses, metadata, status identifiers, or summary information; they do not write to the model parameter file, trigger optimizer updates, move the data read cursor, submit the formal checkpoint list, or change the formal training output.

[0144] S45. Based on the detection response result of the restricted detection action, perform response matching on the candidate task state to obtain the task state identification result.

[0145] The restricted detection action is executed to obtain a detection response result; the response type of the detection response result is identified to obtain a GPU response result, a control communication response result, a data source response result, a checkpoint storage response result, or a worker node control response result; the detection response result is matched with the expected response features corresponding to the candidate states to be distinguished to obtain a response matching result; the target task state is determined from the candidate states to be distinguished based on the response matching result to obtain a task state identification result.

[0146] Specifically, when GPU response detection is normal and training communication has not yet reached the start condition, it is determined to be a normal long-term computation state; when control communication is normal but the training communication sequence number does not advance, it is determined to be a stagnation state of ensemble communication; when data source metadata detection is abnormal and data reading evidence is missing, it is determined to be a data source waiting state; when worker node heartbeat, control response, and GPU events all stop updating, it is determined to be a worker node disconnected state; when checkpoint shards are missing or shard digests are abnormal, it is determined to be a checkpoint sharding abnormal state; when checkpoint shards are complete but list submission evidence is missing, it is determined to be a checkpoint list abnormal state.

[0147] The task state identification result is generated based on the current evidence frontier, candidate task states, state differentiation results, restricted probe actions, and probe response results. Specifically, the target task state is determined from multiple candidate task states through state evidence parsing, candidate state matching, discrimination determination, difference evidence analysis, restricted probe action generation, and response matching processing. The task state identification result includes at least one of the following: normal long-term computation state, normal gradient synchronization state, normal checkpoint upload state, data source waiting state, aggregate communication stagnation state, worker node disconnection state, checkpoint sharding anomaly state, and checkpoint list anomaly state. The task state identification result provides a basis for subsequent anomaly origin location and recovery boundary determination. Subsequent steps search for the first missing evidence, the first anomalous evidence, or the first interrupted dependency relationship in the task state evidence chain based on the task state identification result.

[0148] S5, such as Figure 4 As shown, the task state evidence chain is restored based on the task state identification result to obtain a consistent restoration boundary result.

[0149] In this embodiment, the recovery boundary is not determined directly based on the time of the most recent checkpoint, but rather based on the task state evidence chain to locate the abnormal starting point, backtrack the candidate section, verify the consistency of the training state, and screen the recovery availability.

[0150] S51. Based on the task status identification result, locate the abnormal starting point of the task status evidence chain to obtain abnormal starting point evidence.

[0151] Specifically, the anomaly type is determined based on the task status identification result; the first missing evidence, the first abnormal evidence, or the first interrupted dependency relationship is searched from the task status evidence chain according to the anomaly type; and the task status evidence unit corresponding to the first missing evidence, the first abnormal evidence, or the first interrupted dependency relationship is determined as the anomaly starting point evidence.

[0152] S52. Based on the anomaly initiation evidence, backtrack the task state evidence chain to obtain a candidate recovery aspect set. Determine the corresponding task state evidence chain and backtracking start range based on the anomaly initiation evidence. Starting from the backtracking start range, trace backward along the execution dependencies in the task state evidence chain, filtering task state evidence units that were completed before the anomaly occurred and have complete dependencies. Divide the selected task state evidence units into aspects based on their corresponding task process positions to obtain candidate recovery aspects. Organize the candidate recovery aspects into a set to obtain a candidate recovery aspect set.

[0153] S521. Locate the link position of the abnormal starting point evidence in the task status evidence chain to obtain the starting link position; specifically, extract the task number, work node number, task stage identifier, and task process identifier from the abnormal starting point evidence; determine the corresponding task status evidence chain based on the task number, determine the corresponding node chain segment based on the work node number, and determine the corresponding evidence node based on the task stage identifier and task process identifier; take the position of the evidence node in the task status evidence chain as the starting link position.

[0154] S522. Based on the starting link position, perform reverse dependency tracing on the task state evidence chain to obtain a backtracking evidence path; specifically, performing reverse dependency tracing on the task state evidence chain includes: starting from the starting link position, tracing forward along the preceding event summary, intra-node dependencies, inter-node dependencies, checkpoint fragmentation relationships, and recovery dependency relationships; when an evidence node is found that the task phase is completed, the dependency relationship is closed, and the task process is clearly identified, it is included in the backtracking evidence path; when an evidence node is found that the preceding evidence is missing, the dependency relationship is broken, or the checkpoint fragmentation is incomplete, the backtracking of the corresponding path is terminated.

[0155] S523. Perform recovery node screening on the backtracking evidence path to obtain candidate recovery nodes; including: determining whether the task status evidence units in the backtracking evidence path meet the conditions of task phase completion, complete execution dependency, clear task process identification, and determinable recovery status; and determining the task status evidence units that meet the conditions as candidate recovery nodes.

[0156] S524. Based on the candidate recovery nodes, divide the section range to obtain candidate recovery sections; identify the range according to the node affiliation information and task process information corresponding to the candidate recovery nodes, and divide the candidate recovery nodes with the same range into the same candidate recovery section.

[0157] S525. The candidate recovery sections are aggregated to obtain a candidate recovery section set. Multiple candidate recovery sections are categorized according to checkpoint generation, training steps, and anomaly impact range; the categorized candidate recovery sections are sorted from near to far according to recovery progress; the sorted candidate recovery sections are aggregated to obtain a candidate recovery section set.

[0158] S53. Perform training state consistency verification on the candidate recovery aspect set to obtain state consistency verification results. Extract state elements for each candidate recovery aspect in the candidate recovery aspect set to obtain model parameter state, optimizer state, random number state, data read cursor, checkpoint shard state, checkpoint list state, and output submission state; perform consistency verification on the state elements to determine whether the model parameters correspond to the same training step, whether the optimizer state corresponds to the same update sequence number, whether the random number state summary has been saved, whether the data read cursor corresponds to the training step, whether the checkpoint shard is complete and the summary verification passes, whether the checkpoint list has been submitted or can be regenerated from the complete shard, and whether there is a risk of duplicate submission in the output submission state; obtain the state consistency verification result based on the verification results.

[0159] S54. Based on the state consistency verification result, perform recovery availability filtering on the candidate recovery aspect set to obtain an available recovery aspect set.

[0160] Candidate recovery aspects that pass the state consistency check are identified as usable recovery aspects; candidate recovery aspects with inconsistent model parameters, inconsistent optimizer states, mismatched data read cursors, missing checkpoint shards, incorrect checkpoint lists, or uncertain output submission states are removed; for candidate recovery aspects with complete checkpoint shards but lacking only checkpoint list submission confirmation, if the list can be regenerated based on the complete shards, they are identified as repairable recovery aspects and included in the set of usable recovery aspects.

[0161] S55. Optimize the recovery progress of the available recovery section set to obtain the target recovery section.

[0162] The available recovery sections in the set of available recovery sections are sorted by recovery progress to obtain the recovery progress sorting result; the influence range of the recovery progress sorting result is compared to obtain the recovery influence range result; the available recovery section that is closest to the abnormal starting point, has the highest training progress, and has the smallest recovery influence range is determined as the target recovery section.

[0163] S56. Based on the target recovery section, perform boundary restoration organization to obtain a consistent boundary restoration result.

[0164] Based on the target recovery section, extract the recovery training steps, recovery optimizer update sequence number, recovery data read cursor, recovery random number state summary, recovery checkpoint generation, recovery worker node range, recovery parallel group range, and anomaly checkpoint isolation information; organize the information in a structured manner to obtain consistent recovery boundary results.

[0165] The consistent recovery boundary result is based on task state identification results, evidence of anomaly initiation, a set of candidate recovery aspects, state consistency verification results, and target recovery aspect generation. Specifically, through anomaly initiation location, candidate aspect backtracking, training state consistency verification, recovery availability filtering, recovery progress optimization, and recovery boundary organization processing, the boundary position where the cloud model task can be safely recovered is determined. The consistent recovery boundary result includes the number of recovery training steps, recovery optimizer update sequence number, recovery data read cursor, recovery random number state summary, recovery checkpoint generation, recovery worker node range, recovery parallel group range, and anomaly checkpoint isolation information. The consistent recovery boundary result provides a basis for subsequent anomaly handling, including the recovery initiation point, recovery range, and checkpoint availability. Subsequent steps determine whether to continue waiting, reread data, rebuild the communication group, replace worker nodes, repair the checkpoint list, or execute task recovery based on the consistent recovery boundary result.

[0166] S6. Based on the task status identification results and consistent recovery boundary results, perform anomaly handling to obtain the model task anomaly handling results.

[0167] Specifically, when the task status identification result is normal long-term computation, normal checkpoint upload, or normal model re-sharding, the anomaly handling action is to continue monitoring the corresponding status evidence and not to restart the task; when the task status identification result is data source waiting, the anomaly handling action is to re-initiate reading from the data read cursor corresponding to the consistent recovery boundary result or switch the data source; when the task status identification result is set communication stall, the anomaly handling action is to rebuild the communication group or replace the communication-abnormal worker node, and restore the affected parallel group from the recovery aspect corresponding to the consistent recovery boundary result; when the task status identification result is worker node disconnection, the anomaly handling action is to request a replacement worker node and load the corresponding model shard; when the task status identification result is checkpoint sharding anomaly, the anomaly handling action is to isolate incomplete checkpoint generations and restore from the target recovery aspect; when the task status identification result is checkpoint list anomaly, the anomaly handling action is to regenerate the checkpoint list based on the complete shards.

[0168] The continuity verification includes verification of training step continuity, optimizer update sequence number continuity, data read cursor continuity, random number state recovery, communication sequence number consistency, checkpoint generation continuity, and output submission state. If the continuity verification passes, a model task anomaly handling result is generated; if the continuity verification fails, a recovery failure flag is generated, and an anomaly fingerprint is generated based on the anomaly type, anomaly node, anomaly origin evidence, and anomaly handling action.

[0169] The model task anomaly handling results include task state identification results, consistent recovery boundary results, anomaly handling actions, post-recovery task progress, anomaly checkpoint isolation information, recovery verification results, and duplicate anomaly fingerprints. These results are derived from the task state evidence chain, the current evidence frontier, restricted probe response results, and consistent recovery boundary results, rather than being generated solely based on training steps, heartbeats, or the most recent checkpoint. These results guide actions such as continuing to wait for cloud-based model task execution, switching data sources, rebuilding communication groups, replacing worker nodes, repairing checkpoint lists, isolating anomaly checkpoints, or resuming tasks. By generating these results, the existing technology addresses the difficulty in distinguishing between normal, prolonged periods of stagnation and abnormal standstill, as well as the tendency to recover from incomplete checkpoints or regress indiscriminately after an anomaly. Therefore, this invention enables accurate state identification, consistent determination of recovery boundaries, and matching of anomaly handling actions when a cloud-based model task is stagnant, improving the reliability and continuity of cloud-based model task anomaly handling.

[0170] Furthermore, in order to illustrate the processing effect of the anomaly handling method described in this invention compared with the prior art, a comparison was made between the prior art and the present invention in terms of state recognition accuracy, ability to detect hidden stagnation, accuracy of recovery boundary determination, ability to identify checkpoint anomalies, ability to ensure recovery continuity, and ability to suppress repeated faults, and the above performance comparison results were obtained.

[0171] As can be seen from the simulation comparison results above, compared with the existing technology, the present invention exhibits superior performance in terms of state recognition accuracy, latent stagnation detection capability, recovery boundary determination accuracy, checkpoint anomaly discrimination capability, recovery continuity assurance capability, and repeated fault suppression capability. Specifically, the present invention improves the accuracy of cloud-based model task state recognition by associating and identifying the model computation, data reading, cross-node communication, and checkpoint processing processes through task state evidence chains and current evidence frontiers; furthermore, it enhances the latent stagnation detection capability and checkpoint anomaly discrimination capability by distinguishing between normal long-term operating states and computation, communication, data, and checkpoint anomalies through restricted detection actions; simultaneously, the present invention determines a consistent recovery boundary based on model parameters, optimizer state, random number state, data reading cursor, checkpoint state, and output submission state, enabling anomaly handling actions to match the task state and recovery boundary, thereby improving the accuracy of recovery boundary determination and recovery continuity assurance capability, and reducing the risk of repeated faults or ineffective recovery.

[0172] To further disclose the formation process of the aforementioned state recognition accuracy, latent stagnation detection capability, recovery boundary determination accuracy, checkpoint anomaly detection capability, recovery continuity assurance capability, and recurring fault suppression capability, a cloud-based large-scale model fine-tuning simulation environment with eight working nodes was established. Working nodes W1 to W8 are configured with GPUs, training containers, and local event collectors, respectively. The eight working nodes form the same communication group. The training task is executed in data parallelism, with a total training step count set to 100,000 steps, and a checkpoint is generated every 200 steps. Each checkpoint includes eight model parameter slices, optimizer state, random number state summary, data read cursor, checkpoint slice summary, and checkpoint list. Object storage is used to store checkpoint slices and lists, and the task control plane is used to store task numbers, node change records, communication group numbers, and anomaly handling records.

[0173] During the simulation, task configuration data, worker node status data, training progress data, data reading data, communication process data, checkpoint process data, and resource operation data are collected according to the data structure disclosed in the specification. Events such as data reading, forward computation, backward computation, gradient synchronization, optimizer update, checkpoint shard upload, checkpoint list submission, and model re-sharding are organized into a task status evidence chain. Existing comparative methods use heartbeat detection, GPU utilization, training step update time, and checkpoint retention time as fixed thresholds for judgment. When the training step count does not increase for 120 consecutive seconds or the worker node heartbeat is interrupted, the workgroup is restarted, and the most recent checkpoint is directly loaded from object storage. No joint consistency verification is performed on model parameters, optimizer status, random number status, data reading cursor, checkpoint sharding, and list status.

[0174] The state recognition accuracy test set up 1000 independent task state windows, including 100 each for normal long-duration computation, normal gradient synchronization, normal checkpoint upload, normal model re-sharding, data source waiting, GPU computation anomaly, ensemble communication stagnation, worker node disconnection, checkpoint sharding anomaly, and checkpoint list anomaly. Each window pre-recorded the actual task state, and the method of this invention and the prior art comparison method were run under the same event arrival order, resource response, and network latency conditions. A correct recognition was recorded when the system output state matched the preset actual state. This invention correctly recognized 950 state windows, with a state recognition accuracy of 95%; the prior art correctly recognized 580 state windows, with a state recognition accuracy of 58%.

[0175] The latent stagnation detection capability test included 400 test scenarios where the external behavior remained online but the task process had stopped. These included 120 scenarios of communication blockage, 100 scenarios of GPU computation flow anomalies, 100 scenarios of object storage request lingering, and 80 scenarios where the worker node control plane was online but the training process was disconnected. All these scenarios maintained continuous updates of partial heartbeats, GPU utilization, or network traffic to simulate latent stagnation that is difficult to identify in a timely manner using fixed threshold methods. A successful detection was defined as the system identifying the true anomaly type within 90 seconds and not misclassifying normal long-duration computation, normal gradient synchronization, or normal checkpoint upload as an anomaly. This invention effectively detected 348 scenarios, achieving a latent stagnation detection capability of 87%; existing technologies effectively detected 192 scenarios, achieving a latent stagnation detection capability of 48%.

[0176] The accuracy test for determining the recovery boundary involved setting up 300 anomalous tasks requiring recovery. Each task had three pre-defined candidate recovery sections: the most recent but incomplete section, an earlier section with the same state, and an even earlier complete section. The recovery boundary was considered correctly determined when the selected target recovery section simultaneously met the following conditions: model parameters corresponding to the same training step, optimizer state corresponding to the same update sequence number, random number state summary saved, data read cursor corresponding to the training step, checkpoint slice complete and summary verification passed, and the selected section was the highest training progress position meeting these conditions. This invention correctly determined the target recovery section for 279 tasks, with an accuracy of 93%; existing technologies, selecting based on the most recent checkpoint time, only correctly selected the section for 150 tasks, with an accuracy of 50%.

[0177] The checkpoint anomaly detection capability test included 200 checkpoint anomaly scenarios: 60 missing fragments, 50 inconsistent fragment digests, 50 complete fragments but not submitted manifests, and 40 fragments with incorrect manifest references. A correct detection was achieved when the system could distinguish between unusable fragment anomalies and repairable anomalies that could be regenerated and output the corresponding isolated checkpoint generation or manifest repair action. This invention correctly detected 180 scenarios, achieving a 90% checkpoint anomaly detection capability. Existing technologies primarily rely on checkpoint generation time and the number of objects for detection, correctly detecting 110 scenarios, achieving a 55% capability.

[0178] The continuity assurance capability test included 250 anomaly recovery tasks. After recovery, 50 training steps were executed consecutively, and the training step count, optimizer update sequence number, data read cursor, random number status, communication sequence number, checkpoint generation, and output submission status were verified. A continuity verification was considered successful when the number of training steps continuously increased, data samples were not read repeatedly, the optimizer status was not misaligned, the checkpoint generation did not regress, and the results were not resubmitted. This invention passed all verifications with 230 successful recoveries, achieving a continuity assurance capability of 92%; the prior art passed all verifications with 135 successful recoveries, achieving a continuity assurance capability of 54%.

[0179] The test for suppressing recurring anomalies included 200 scenarios: 70 scenarios of continuous blocking of the same communication node, 60 scenarios of repeated selection of the same incomplete checkpoint, 40 scenarios of repeated failed list submissions due to the same object storage failure, and 30 scenarios of repeated task restarts due to timeouts from the same data source. If the system can generate an anomaly fingerprint based on the anomaly type, anomaly node, evidence of the anomaly's origin, and anomaly handling actions, and avoid repeatedly loading isolated checkpoints, repeating invalid recovery actions, or limiting the handling scope to affected nodes and parallel groups when the same anomaly reappears, then the recurring anomaly is considered suppressed. This invention suppressed repeated invalid handling in 174 scenarios, achieving a recurring anomaly suppression capability of 87%; existing technologies avoided repeated handling in 90 scenarios, achieving a suppression capability of 45%.

[0180] Taking a ensemble communication stall and recovery process as an example, when the training task reached step 18640, worker nodes W1 to W8 had all generated evidence of backpropagation completion and entered the gradient synchronization stage corresponding to communication sequence number 9320. Among them, the control plane heartbeat, GPU utilization, and network reception of W1 to W4 and W6 to W8 were still updating, and the control plane heartbeat of W5 was also normal. However, none of the eight worker nodes generated evidence of communication completion, and the training steps did not advance for 70 consecutive seconds. The current evidence front shows that all nodes are stuck at the same communication sequence number. The candidate task states include the normal gradient synchronization state and the ensemble communication stall state. The matching values ​​of the two states are 0.78 and 0.74, respectively. The difference in matching values ​​of 0.04 is lower than the preset discrimination threshold of 0.10, thus generating a state ambiguity result. The preset state discrimination threshold is used to determine the lower limit of discrimination of the candidate state to determine whether there is ambiguity. The typical value is 0.10, which is calibrated according to the degree of impact of state misjudgment on anomaly handling.

[0181] To address the aforementioned state ambiguity, the system extracts training communication sequence number advancement evidence and control communication response evidence from a preset task state template as discrepancy evidence items, and generates a communication control group connectivity query that does not change model parameters, optimizer state, gradient values, or data read cursors. The detection results show that W1 to W4 and W6 to W8 all returned communication control responses, while W5 did not return a response within the specified time. Based on this, the response matching value for the ensemble communication stagnation state was increased to 0.93, and the response matching value for the normal gradient synchronization state was decreased to 0.31, ultimately identifying W5 as the cause of the ensemble communication stagnation. Existing technology, because W5's heartbeat and GPU utilization are still updating, cannot determine the anomaly before the training steps reach a fixed timeout of 120 seconds, and directly restarts all eight worker nodes after the timeout.

[0182] The system uses the missing evidence of the first communication response corresponding to W5 as the starting point of the anomaly. It then backtracks along intra-node dependencies, inter-node communication dependencies, and checkpoint shard associations, obtaining three candidate recovery aspects: checkpoint generations C18600, C18400, and C18200. Verification results show that although C18600 is the most recent, the model parameter shard corresponding to W5 is missing, and the checkpoint list is not formed; therefore, it is judged as an unusable recovery aspect. C18400 has eight model parameter shard summaries that pass verification, the optimizer update sequence number is 18400, the data read cursor is 3680000, and the random number state summary and checkpoint list are complete. C18200 is also complete, but its training progress is lower than C18400. Therefore, the system identifies C18400 as the target recovery aspect and limits the scope of the recovery working nodes to W5 and its communication group.

[0183] During anomaly handling, the system isolates checkpoint generation C18600, requests a replacement worker node W5', loads the corresponding model shards, optimizer status, random number status, and data read cursor for C18400, rebuilds the communication group, and continues training. After recovery, 50 training steps are executed consecutively, with the training step number continuously advancing from 18401 to 18450. The optimizer update sequence number is synchronously incremented, and the data read cursor continuously advances from 3680000. There are no instances of duplicate sample reads, checkpoint generation regression, or duplicate result submissions, and the continuity verification is successful. If C18600 is subsequently selected again or the same communication blockage is detected with W5, the system directly excludes the isolated checkpoint based on the anomaly fingerprint and prioritizes replacing the corresponding node, thereby avoiding repeated and ineffective recovery.

[0184] Taking the checkpoint list anomaly as another example, the eight model parameter slices, optimizer state, random number state, and data read cursor of checkpoint generation C18800 have all been written to object storage and the digest verification has passed, but the checkpoint list submission evidence is missing due to a timeout in the list submission request. This invention identifies this as a checkpoint list anomaly rather than a checkpoint slice anomaly, and continues to use C18800 after regenerating the list based on the complete slices, without needing to roll back to C18600. Existing technologies only detect incomplete recent checkpoint records, directly abandoning C18800 and rolling back to an earlier checkpoint, resulting in the loss of completed training progress. The above statistical results and specific handling process show that all six performance data can be repeatedly obtained from clearly defined test samples, judgment conditions, and calculation processes.

[0185] Those skilled in the art should understand that the cloud-based model task state recognition-based anomaly handling method described in this invention can be deployed in various cloud computing environments such as cloud GPU clusters, distributed training platforms, and model fine-tuning service nodes. It can be implemented using a pure hardware architecture, a pure software architecture, or a hardware-software integrated architecture. This invention can be prepared as a computer program product and stored in various non-volatile computer-readable storage media, including standardized storage carriers such as solid-state drives, flash memory chips, mobile storage devices, optical discs, and cloud storage servers.

[0186] The above embodiments illustrate in detail the implementation of the present invention in cloud-based large-scale model training and maintenance, and distributed task anomaly handling scenarios. Those skilled in the art should understand that adjustments can be made to the implementation form and details for different model sizes, cluster architectures, and business objectives without departing from the core logic of the present invention. The scope of protection of the present invention is defined by the claims and their equivalent technical solutions.

Claims

1. A cloud-based model task state recognition-based anomaly handling method, characterized in that, include: Obtain the runtime data of the cloud-based model tasks and aggregate the task process to obtain the basic data of the model tasks; The basic data of the model task are organized into an evidence chain to obtain the task status evidence chain; Based on the preset task status template, the evidence frontier of the task status evidence chain is extracted to obtain the current evidence frontier; Based on the current evidence frontier, state identification is performed to obtain the task state identification result; Based on the task state identification result, the task state evidence chain is restored to determine the boundary, and a consistent restoration boundary result is obtained. Anomaly handling is performed based on the task state identification results and consistent recovery boundary results to obtain the model task anomaly handling results.

2. The anomaly handling method according to claim 1, characterized in that, The basic data of the model task are organized into an evidence chain to obtain a task state evidence chain, including: The basic data of the model task are parsed to obtain a set of task process events; Based on the preset task stages, the set of task process events is state-mapped to obtain a set of state events. The set of state events is processed into evidence units to obtain task state evidence units; Based on the execution dependencies between the task status evidence units, association tags are performed to obtain status evidence association results; Based on the association results of the state evidence, a chain is formed to obtain the task state evidence chain.

3. The anomaly handling method according to claim 2, characterized in that, Associative tagging is performed based on the execution dependencies between the task status evidence units, including: Node attribution identification is performed on the task status evidence unit to obtain the node evidence attribution result; Based on the node evidence attribution results, the task status evidence units within the same working node are sequentially matched to obtain the intra-node dependency relationship. Based on the task phase transition relationship, the task status evidence units between different working nodes are collaboratively matched to obtain the inter-node dependency relationship. The consistency of the intra-node dependencies and inter-node dependencies is verified to obtain the valid dependencies. The valid dependencies are associated and marked to obtain the state evidence association results.

4. The anomaly handling method according to claim 3, characterized in that, Consistency checks are performed on the intra-node dependencies and inter-node dependencies to obtain valid dependencies, including: The training progress continuity is verified for the dependencies within the nodes to obtain the verification results within the nodes. Perform a coordination phase consistency check on the dependencies between the nodes to obtain the node check results; Based on the intra-node verification results and inter-node verification results, task process consistency matching is performed to obtain dependency matching results. The execution order validity is filtered based on the dependency matching results to obtain valid dependencies.

5. The anomaly handling method according to claim 1, characterized in that, Based on a preset task state template, the evidence frontier of the task state evidence chain is extracted to obtain the current evidence frontier, including: The preset task state template is parsed to obtain a set of template evidence items; Based on the set of template evidence items, the task status evidence chain is matched to obtain the evidence matching result; The evidence matching results are marked with evidence arrival information to obtain evidence arrival results; Based on the arrival results of the aforementioned evidence, the latest valid evidence for each working node is screened to obtain the node evidence frontier; The current evidence front is obtained by aligning the node evidence front with the task phase.

6. The anomaly handling method according to claim 5, characterized in that, Based on the arrival results of the aforementioned evidence, the latest valid evidence for each working node is screened, including: The results of the evidence arrival are categorized into working nodes to obtain a node evidence set; Arrival status identification is performed on the node evidence set to obtain the evidence of arrived nodes; The validity of the arrived node evidence is determined to obtain valid node evidence. Based on the valid node evidence, the state advancement position is filtered to obtain the latest valid evidence; The latest valid evidence corresponding to each working node is organized into a frontier to obtain the node evidence frontier.

7. The anomaly handling method according to claim 1, characterized in that, Based on the current evidence frontier, state identification is performed to obtain task state identification results, including: The current evidence frontier is analyzed using state evidence analysis to obtain frontier evidence features; Candidate task states are obtained by matching the frontier evidence features with candidate states based on a preset task state template. The candidate task states are distinguished to obtain state distinction results; Based on the state differentiation results, a difference evidence analysis is performed on the candidate task states to obtain restricted detection actions; Based on the detection response results of the restricted detection action, the candidate task states are matched accordingly to obtain the task state identification results.

8. The anomaly handling method according to claim 7, characterized in that, Based on the state differentiation results, differential evidence analysis is performed on the candidate task states to obtain restricted detection actions, including: The state distinction results are used to filter candidate states to obtain candidate states to be distinguished; Based on the preset task state template, expected evidence is extracted from the candidate states to be distinguished to obtain candidate state evidence items. The candidate state evidence items are compared for evidence differences to obtain the difference evidence items; Based on the difference evidence items, the detection type is matched to obtain candidate detection actions; The candidate detection actions are subjected to training state influence constraints to obtain restricted detection actions.

9. The anomaly handling method according to claim 1, characterized in that, Based on the task state identification result, the task state evidence chain is restored to determine the boundary, resulting in a consistent restoration boundary result, including: Based on the task status identification result, the abnormal starting point of the task status evidence chain is located to obtain abnormal starting point evidence. Based on the abnormal starting point evidence, the candidate aspect backtracking is performed on the task state evidence chain to obtain a set of candidate recovery aspects. The training state consistency check is performed on the candidate recovery section set to obtain the state consistency check result. Based on the state consistency verification results, the set of candidate recovery aspects is filtered for recovery availability to obtain a set of available recovery aspects. The recovery progress of the available recovery sections is optimized to obtain the target recovery section; Based on the target restoration section, the restoration boundary organization is performed to obtain a consistent restoration boundary result.

10. The anomaly handling method according to claim 9, characterized in that, Based on the abnormal starting point evidence, the task state evidence chain is backtracked to obtain a set of candidate recovery aspects, including: The link position of the abnormal starting point evidence in the task status evidence chain is located to obtain the starting point link position; Based on the starting link position, reverse dependency tracing is performed on the task status evidence chain to obtain the backtracking evidence path; The backtracking evidence path is filtered to obtain candidate recovery nodes; Based on the candidate recovery nodes, the sectional range is divided to obtain the candidate recovery section; The candidate recovery sections are organized into a set to obtain a set of candidate recovery sections.

Images