Distributed training fault processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a distributed training fault processing method and device, computer equipment, a computer readable storage medium and a computer program product. A distributed training task is issued to the distributed training cluster, and training logs, task breakpoint data and model related information generated in the training process are shared and stored in real time; monitoring the training log in real time based on the fault detection large model, and generating and feeding back a model training progress, model verification precision and training suggestions to the deployment platform under the condition of judging that no abnormal information exists in the training log; when it is judged that the abnormal information exists in the training log, abnormal information and a processing scheme are generated according to the abnormal type and fed back to the deployment platform; and executing a corresponding processing operation based on an operable instruction in the processing scheme, and associating the exception information and the processing scheme with the task breakpoint data in the shared storage so as to provide data support during exception processing and task restarting. The method can improve the effective utilization rate of training resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and in particular to a distributed training fault processing method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of deep learning, the demand for models of artificial intelligence is increasing, and the emergence of large models can better meet the high requirements of artificial intelligence.

[0003] However, in the process of training a large model, the distributed resources used are more than those of a small model, and the computing power cost required for training a large model also increases sharply. The stability of the connection between machines directly determines the stability and performance of task training. When any link fails, the entire task training will fail, resulting in great waste of resources and a decrease in effective utilization. Therefore, how to efficiently utilize resources for iteration in a shorter time is a problem that needs to be solved at present. SUMMARY

[0004] Therefore, it is necessary to provide a distributed training fault processing method, device, computer equipment, computer readable storage medium and computer program product capable of improving the effective utilization of large model training resources to solve the above technical problems.

[0005] In a first aspect, the present application provides a distributed training fault processing method, comprising:

[0006] Distributing a distributed training task to a distributed training cluster, and performing real-time shared storage of training logs, task breakpoint data and model-related information generated in the training process;

[0007] Reading the training logs in the shared storage according to a preset period, inputting the training logs into a fault detection large model, generating a model training progress, a model verification accuracy and a training suggestion in the case that the fault detection large model determines that there is no abnormal information in the training logs, and generating abnormal information and a processing scheme according to the abnormal type in the case that the fault detection large model determines that there is abnormal information in the training logs;

[0008] In the case that the model training progress, the model verification accuracy and the training suggestion are obtained, the training progress, the model verification accuracy and the training suggestion are fed back to a deployment platform for user viewing; in the case that the abnormal information and the processing scheme are obtained, the abnormal information is fed back to the deployment platform for user viewing, corresponding processing operations are performed based on the operable instructions in the processing scheme, and the abnormal information and the processing scheme are associated with the task breakpoint data in the shared storage to provide data support for abnormal processing and task restart.

[0009] In one of the embodiments, the configuration of the shared storage and the read-write process include:

[0010] The bandwidth threshold and the storage space threshold of the shared storage are pre-configured;

[0011] The process read-write coordination mechanism is configured for the shared storage, and the read-write requests of multiple processes for training logs, task breakpoint data and model-related information in the distributed training process are queued and scheduled through file locking or process scheduling priority allocation.

[0012] In one of the embodiments, the abnormal information and the processing scheme are generated according to the abnormal type, including:

[0013] In the case of detecting node failure, the node abnormal features are extracted from the training log to generate a special error code and a processing scheme for node isolation and task retransmission; the abnormal node is located according to the special error code and the isolation operation is performed, and the task breakpoint data in the shared storage is called to reissue the training task to the distributed training cluster to pull up the task based on the breakpoint;

[0014] In the case of detecting business failure and the business failure being a code level failure, a processing scheme containing code correction suggestions and path adjustment scheme is generated, the processing scheme is fed back to the deployment platform for user to view, and the training log, task breakpoint data and model-related information in the shared storage are reserved for subsequent troubleshooting; wherein, the code level failure includes code writing error and path error, and the fault detection large model extracts the error associated code location and path information from the training log;

[0015] In the case of detecting unknown failure without explicit solution, the node running parameters and the failure triggering time are shared to provide data support for manual troubleshooting.

[0016] In one of the embodiments, the method further includes:

[0017] In the case of detecting unknown failure without explicit solution, if a processing scheme is provided by manual troubleshooting, the related training log, task breakpoint data, model-related information, node running parameter and failure triggering time in the shared storage are taken as input, and the manually provided processing scheme is taken as output, and the fault detection large model is iteratively trained.

[0018] In one of the embodiments, the pre-deployment and pre-training detection process of the distributed training cluster includes:

[0019] The unified registration and management of the cluster nodes are completed through the management container platform, the network communication protocol between nodes, the storage connection interface and the card-to-card data transmission parameters are configured;

[0020] Start the pre-deployed network, storage, inter-card communication fault monitoring tool, respectively, to detect the network connectivity of the cluster node, the storage read-write response speed, and the inter-card data transmission success rate item by item;

[0021] If the detection results all meet the preset standard, it is confirmed that the distributed training cluster has the task carrying capacity; if there is a detection item that does not meet the preset standard, the abnormal cluster node and the specific fault point are located through the fault monitoring tool, the detection is restarted after the repair operation is performed, and all detection items meet the standard.

[0022] In one of the embodiments, after performing the corresponding processing operation based on the operable instructions in the processing scheme, it further includes:

[0023] Real-time monitoring of the running state of the corresponding training task in the distributed training cluster, and checking the update of the training log in the shared storage and the integrity of the task breakpoint data, and determining whether the task is restored to normal operation;

[0024] If the task is restored to normal operation, continue to read the training log in the shared storage according to the preset period, and input it to the fault detection large model to maintain the monitoring of the training process;

[0025] If the task is not restored to normal operation, send a manual troubleshooting notification to the operation and maintenance personnel through the deployment platform, the manual troubleshooting notification includes fault key information, executed operation record, and training log and task breakpoint data in the shared storage; after completing the fault troubleshooting, trigger the restart instruction to call the task breakpoint data in the shared storage, and reissue the training task to the distributed training cluster to make the fault detection large model restore the monitoring and inference process of the training log.

[0026] In a second aspect, the application also provides a distributed training fault processing device, which includes:

[0027] The issuing module is configured to issue a distributed training task to a distributed training cluster, and to perform real-time shared storage of training logs, task breakpoint data, and model-related information generated during the training process.

[0028] The determination module is configured to read the training log in the shared storage according to a preset period, and input the training log into the fault detection large model; in the case that the fault detection large model determines that there is no abnormal information in the training log, generate the model training progress, the model verification accuracy, and the training suggestion; in the case that the fault detection large model determines that there is abnormal information in the training log, generate the abnormal information and the processing scheme according to the abnormal type;

[0029] a feedback module, configured to feed the training progress, the model verification accuracy and the training suggestion to the deployment platform for the user to view in the case of obtaining the model training progress, the model verification accuracy and the training suggestion; feed the abnormal information to the deployment platform for the user to view in the case of obtaining the abnormal information and the processing scheme, perform the corresponding processing operation based on the operable instruction in the processing scheme, and associate the abnormal information and the processing scheme with the task breakpoint data in the shared storage to provide data support for abnormal processing and task restart.

[0030] In a third aspect, the present application also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0031] issuing a distributed training task to the distributed training cluster, and performing real-time shared storage on training logs, task breakpoint data and model related information generated in the training process;

[0032] reading the training logs in the shared storage according to a preset period, inputting the training logs into the fault detection large model, generating the model training progress, the model verification accuracy and the training suggestion in the case of no abnormal information in the training logs determined by the fault detection large model, and generating the abnormal information and the processing scheme according to the abnormal type in the case of abnormal information in the training logs determined by the fault detection large model;

[0033] feeding the training progress, the model verification accuracy and the training suggestion to the deployment platform for the user to view in the case of obtaining the model training progress, the model verification accuracy and the training suggestion; feeding the abnormal information to the deployment platform for the user to view in the case of obtaining the abnormal information and the processing scheme, performing the corresponding processing operation based on the operable instruction in the processing scheme, and associating the abnormal information and the processing scheme with the task breakpoint data in the shared storage to provide data support for abnormal processing and task restart.

[0034] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0035] issuing a distributed training task to the distributed training cluster, and performing real-time shared storage on training logs, task breakpoint data and model related information generated in the training process;

[0036] reading the training logs in the shared storage according to a preset period, inputting the training logs into the fault detection large model, generating the model training progress, the model verification accuracy and the training suggestion in the case of no abnormal information in the training logs determined by the fault detection large model, and generating the abnormal information and the processing scheme according to the abnormal type in the case of abnormal information in the training logs determined by the fault detection large model;

[0037] In the case where the model training progress, model validation accuracy and training suggestion are obtained, the training progress, model validation accuracy and training suggestion are fed back to the deployment platform for the user to view; in the case where the abnormal information and the processing scheme are obtained, the abnormal information is fed back to the deployment platform for the user to view, the corresponding processing operation is performed based on the operable instructions in the processing scheme, and the abnormal information and the processing scheme are associated with the task breakpoint data in the shared storage to provide data support for abnormal processing and task restart.

[0038] In the fifth aspect, the present application also provides a computer program product, comprising a computer program which, when executed by a processor, implements the following steps:

[0039] Distributed training tasks are issued to a distributed training cluster, and training logs, task breakpoint data and model related information generated during the training process are stored in real time;

[0040] The training logs in the shared storage are read according to a preset period, and the training logs are input into the fault detection large model; in the case where the fault detection large model determines that there is no abnormal information in the training logs, the model training progress, the model validation accuracy and the training suggestion are generated; in the case where the fault detection large model determines that there is abnormal information in the training logs, the abnormal information and the processing scheme are generated according to the abnormal type;

[0041] In the case where the model training progress, model validation accuracy and training suggestion are obtained, the training progress, model validation accuracy and training suggestion are fed back to the deployment platform for the user to view; in the case where the abnormal information and the processing scheme are obtained, the abnormal information is fed back to the deployment platform for the user to view, the corresponding processing operation is performed based on the operable instructions in the processing scheme, and the abnormal information and the processing scheme are associated with the task breakpoint data in the shared storage to provide data support for abnormal processing and task restart.

[0042] The distributed training fault processing method, device, computer equipment, computer readable storage medium and computer program product can distribute a distributed training task to a distributed training cluster, perform real-time shared storage on training logs, task breakpoint data and model related information generated in a training process, read the training logs in the shared storage according to a preset period, input the training logs into a fault detection large model, generate model training progress, model verification accuracy and training suggestions in the case where the fault detection large model determines that there is no abnormal information in the training logs, generate abnormal information and a processing scheme according to the abnormal type in the case where the fault detection large model determines that there is abnormal information in the training logs, feed back the training progress, model verification accuracy and training suggestions to a deployment platform for a user to view in the case where the model training progress, model verification accuracy and training suggestions are obtained, feed back the abnormal information to the deployment platform for the user to view in the case where the abnormal information and the processing scheme are obtained, perform a corresponding processing operation based on an operable instruction in the processing scheme, and associate the abnormal information and the processing scheme with the task breakpoint data in the shared storage, so as to provide data support for abnormal processing and task restart. The fault detection large model can perform timing monitoring based on the training logs in the shared storage, and in the case where there is abnormal information in the logs, real-time generation of corresponding fault information and a processing scheme can reduce the troubleshooting time of large-scale distributed training abnormalities, thereby improving the efficiency of model training. The scheduling backend can perform a corresponding processing operation based on the operable instruction in the processing scheme, which can optimize the response time of the deployment platform to the training task abnormality, reduce the interruption interval of model training, and thereby improve the effective utilization rate of overall training resources and the running stability of the distributed cluster. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other related drawings without creative labor based on these drawings.

[0044] Figure 1 An application environment diagram of the distributed training fault processing method in an embodiment;

[0045] Figure 2 A flowchart of the distributed training fault processing method in an embodiment;

[0046] Figure 3 A flowchart of the distributed training fault processing method in another embodiment;

[0047] Figure 4A structural block diagram of a processing device for distributed training failure in an embodiment;

[0048] Figure 5 An internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0049] For the purpose, technical solutions and advantages of the present application to be clearer, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0050] It should be noted that the terms "first", "second" and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "a plurality of" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the options or any combination of multiple options.

[0051] The distributed training failure processing method provided by the embodiments of the present application can be applied to the application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through the network. The data storage system can store the data required by the server 104 to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. Specifically, the terminal 102 or the server 104 completes a distributed training failure processing method, which includes:

[0052] Distributed training tasks are issued to a distributed training cluster, and training logs, task breakpoint data and model related information generated in the training process are stored in real time;

[0053] The training logs in the shared storage are read according to a preset period, and the training logs are input to a fault detection large model. In the case that the fault detection large model determines that there is no abnormal information in the training logs, the model training progress, the model verification accuracy and the training suggestion are generated. In the case that the fault detection large model determines that there is abnormal information in the training logs, the abnormal information and the processing scheme are generated according to the abnormal type;

[0054] Once the model training progress, model validation accuracy, and training suggestions are obtained, they are fed back to the deployment platform for user viewing. Once anomaly information and handling solutions are obtained, the anomaly information is fed back to the deployment platform for user viewing. Based on the operable instructions in the handling solution, the corresponding handling operations are executed. The anomaly information and handling solutions are also associated with the task breakpoint data in shared storage to provide data support for anomaly handling and task restart.

[0055] Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0056] In one exemplary embodiment, such as Figure 2 As shown, a method for handling distributed training failures is provided, which can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps 202 to 206. Wherein:

[0057] Step 202: Distribute distributed training tasks to the distributed training cluster and share and store training logs, task breakpoint data and model-related information generated during the training process in real time.

[0058] The distributed training cluster is used to host the training of distributed tasks. Training logs include various program execution events, statuses, and performance metrics during the training process. Task breakpoint data refers to data that allows the training task to be accurately recovered from a specific point in time. Model-related information includes the model's architecture definition, configuration files, and evaluation results.

[0059] For example, the scheduling backend distributes model training tasks, i.e. distributed training tasks, to the distributed training cluster. After the training task starts, the training logs, task breakpoints, and model-related information generated during the model training process are collected in real time and stored in shared storage.

[0060] Step 204: Read the training logs in the shared storage according to the preset cycle and input the training logs into the fault detection big model; if the fault detection big model determines that there is no abnormal information in the training logs, generate the model training progress, model verification accuracy and training suggestions; if the fault detection big model determines that there is abnormal information in the training logs, generate abnormal information and handling solutions according to the abnormality type.

[0061] The fault detection model is primarily used for real-time analysis of training logs and generation of corresponding processing information. Anomaly information includes node fault information, business fault information, and other ambiguous issue information.

[0062] For example, the fault detection big data model reads the training logs in the shared storage at preset intervals and performs real-time analysis on the training logs. If the fault detection big data model determines that there is no abnormal information in the training logs, it sends the status information of normal task training to the scheduling backend and generates model training progress, model validation accuracy, and training suggestions; if the fault detection big data model determines that there is abnormal information in the training logs, it generates abnormal information and corresponding handling solutions according to the abnormality type.

[0063] Step 206: If the model training progress, model validation accuracy, and training suggestions are obtained, the training progress, model validation accuracy, and training suggestions are fed back to the deployment platform for users to view; if the exception information and handling plan are obtained, the exception information is fed back to the deployment platform for users to view, the corresponding handling operation is executed based on the operable instructions in the handling plan, and the exception information and handling plan are associated with the task breakpoint data in the shared storage to provide data support for exception handling and task restart.

[0064] For example, when the model training progress, model validation accuracy, and training suggestions are obtained, the scheduling backend can be used to feed back these information to the deployment platform for users to view; when abnormal information and handling solutions are obtained, the scheduling backend can be used to feed back the abnormal information to the deployment platform for users to view.

[0065] For example, the scheduling backend also executes corresponding node operations, network detection and other processing operations based on the operable instructions in the processing plan, and associates the exception information and processing plan with the task breakpoint data in the shared storage, thereby providing data support for exception handling and task restart.

[0066] In the above-mentioned method for handling distributed training faults, the fault detection model performs periodic monitoring based on training logs in shared storage, and generates corresponding fault information and handling solutions in real time when abnormal information is found in the logs. This reduces the time for troubleshooting large-scale distributed training faults, thereby improving the efficiency of model training. By scheduling the backend to execute corresponding processing operations based on operable instructions in the handling solution, the response time of the deployment platform to abnormal training tasks can be optimized, the interruption interval of model training can be reduced, and the overall effective utilization of training resources and the operational stability of the distributed cluster can be improved.

[0067] In one embodiment, the configuration and read / write process of shared storage includes: pre-configuring the bandwidth threshold and storage space threshold of shared storage; configuring a process read / write coordination mechanism for shared storage, and queuing and scheduling read / write requests from multiple processes for training logs, task breakpoint data and model-related information during distributed training through file locks or process scheduling priority allocation.

[0068] Shared storage is used to store training logs, task breakpoint data, and model-related information generated during distributed training, and is also used to input text information into the fault detection model. File locking is a mechanism that coordinates access to the same file or part of a file by multiple processes to prevent multiple processes from writing to the same file area simultaneously and to ensure that a process performing read / write operations is not interrupted by read / write operations from other processes. Process scheduling priority is a numerical attribute used to determine the next executing process, with processes having higher priority executing read / write operations first.

[0069] For example, the bandwidth and storage space thresholds of the shared storage are pre-configured, so that the shared storage has high bandwidth and large storage space, thereby enabling the shared storage to save task breakpoint data in the distributed training process at a faster speed, while also ensuring that the shared storage has enough space to store the model.

[0070] For example, a process read / write coordination mechanism can be configured for shared storage. By allocating file locks or process scheduling priorities, read / write requests from multiple processes for training logs, task breakpoint data, and model-related information during distributed training can be queued and scheduled. This allows shared storage to avoid multiple processes performing read / write operations simultaneously and makes it less likely for distributed training tasks to trigger deadlock problems during training.

[0071] In this embodiment, by pre-configuring bandwidth and storage space thresholds for shared storage, the shared storage can have faster read and write speeds and larger storage space, thereby effectively improving the utilization rate of distributed training and reducing the occurrence of space shortage errors. By configuring process read and write operations for shared storage, the problem of task interruption during distributed training can be avoided.

[0072] In one embodiment, generating exception information and handling solutions according to exception type includes: when a node failure is detected, extracting node exception features from the training logs, generating a dedicated error code and a handling solution for node isolation and task resending; locating the exception node based on the dedicated error code and performing isolation operations, while simultaneously calling the task breakpoint data in shared storage to resend the training task to the distributed training cluster to restart the task based on the breakpoint; when a business failure is detected, and the business failure is a code-level failure, generating a handling solution including code correction suggestions and path adjustment schemes, feeding the handling solution back to the deployment platform for user viewing, and retaining the training logs, task breakpoint data, and model-related information in shared storage for subsequent investigation; wherein, code-level failures include code writing errors and path errors, and the fault detection model extracts the error-related code location and path information from the training logs; when an unknown failure without a clear solution is detected, sharing the node running parameters and the fault triggering timing to provide data support for manual investigation.

[0073] The anomaly types include node failures, business failures, and other unspecified failures. Node failures refer to hardware or software environment failures that occur during the execution of training tasks; business failures can be code-level failures. Characteristics of node anomalies include training logs not being updated for extended periods, a significant decrease in task training speed, and network communication errors appearing in the training logs.

[0074] For example, when the fault detection model detects a node fault, the fault detection model extracts the node's abnormal features from the training log, generates a special error code, and sends it to the scheduling backend. The scheduling backend locates the abnormal node based on the special error code and performs node isolation. At the same time, it calls the task breakpoint data in the shared storage to resend the training task to the distributed training cluster, thereby re-sending the task based on the breakpoint.

[0075] For example, when the fault detection big model detects a business fault, i.e. a code-level fault, the fault detection big model extracts the code location and path information associated with the error from the training log based on the code-level fault, including code writing errors and path errors, and generates a processing solution that includes code correction suggestions and path adjustment schemes. At the same time, the processing solution is fed back to the deployment platform for users to view, and the training logs, task breakpoint data and model-related information in the shared storage are retained for subsequent investigation.

[0076] For example, when the fault detection big model detects an unknown fault without a clear solution, the fault detection big model summarizes the fault problem information, generates key information such as the node running parameters and fault triggering time of this distributed task, and stores them in shared storage. At the same time, the acquired key information is fed back to the deployment platform to provide data support for manual troubleshooting.

[0077] In this embodiment, the fault detection big model generates corresponding anomaly information and processing solutions based on different anomaly types, and summarizes key log information and fault solutions to feed back to the deployment platform. This can optimize the deployment platform's response time to training task anomalies and improve the efficiency of the fault detection big model in locating and processing fault tasks.

[0078] In one embodiment, the method further includes: when an unknown fault without a clear solution is detected, if a solution is provided through manual investigation, the relevant training logs, task breakpoint data, model-related information, node running parameters, and fault triggering timing in the shared storage are used as inputs, and the manually provided solution is used as output to iteratively train the fault detection large model.

[0079] For example, when the fault detection model detects an unknown fault without a clear solution, the model summarizes and reports the fault information to the deployment platform to provide data support for manual troubleshooting. After manual troubleshooting and the provision of a solution, the model uses the training logs, task breakpoint data, model-related information, node running parameters, and fault triggering timing related to the current training task in the shared storage as inputs to the fault detection model. The manually provided solution is then used as the output of the fault detection model. This process iteratively trains the fault detection model to continuously cover unknown faults and provide multiple solutions corresponding to unknown faults.

[0080] In this embodiment, by iteratively training the fault detection model with the processing scheme corresponding to the unknown fault, the ability of the fault detection model to generate fault solutions can be improved, thereby strengthening the fault handling capability of the large model and optimizing the operation and maintenance efficiency of the large model.

[0081] In one embodiment, the pre-deployment and pre-training testing process of the distributed training cluster includes: completing unified registration and management of cluster nodes through a management container platform, configuring inter-node network communication protocols, storage connection interfaces, and inter-card data transmission parameters; starting the pre-deployed network, storage, and inter-card communication fault monitoring tools to test the network connectivity, storage read / write response speed, and inter-card data transmission success rate of the cluster nodes item by item; if the test results all meet the preset standards, confirming that the distributed training cluster has the ability to carry out tasks; if any test item does not meet the preset standards, locating the abnormal cluster node and specific fault point through the fault monitoring tool, performing repair operations, and restarting the testing until all test items meet the standards.

[0082] A container is a standard, lightweight software unit used to package an application and all its dependencies. A container management platform can be Kubernetes, which includes a control plane and multiple worker nodes for automating the deployment, scaling, and management of containerized applications.

[0083] For example, unified registration and management of cluster nodes is used to enable the cluster control plane to control each worker node. Necessary agents, including a container runtime, kubelet, and kube-proxy, are deployed on each worker node. These agents continuously send resource specifications of each worker node, including CPU, memory, GPU, and network identifiers, to the cluster control plane, thereby completing the registration of each worker node with the cluster control plane. The cluster control plane then adds successfully registered nodes to its resource scheduling pool and continuously monitors these nodes to complete the cluster control plane's management of the cluster nodes.

[0084] For example, by configuring the network communication protocol between cluster nodes through the Container Network Interface (CNI) plugin, containers within the cluster can communicate over the network when located on different cluster nodes, thereby enabling direct data exchange of memory between nodes. By configuring each node in the cluster to access one or more shared storage units and connecting the shared storage to the cluster via the Kubernetes CSI driver to complete the storage connection interface configuration, each training node can read training data from the same shared storage and store the model checkpoints generated during training on the shared storage. Configuring inter-card data communication enables data communication within and between cluster nodes via high-speed interconnect technology and high-performance networks.

[0085] For example, Kubernetes is used to uniformly register and manage the nodes of the distributed training cluster, configure the network communication protocols, storage connection interfaces, and inter-card data transmission parameters between cluster nodes, and complete the pre-deployment of the distributed training cluster. This ensures that the pre-deployed distributed training cluster has fault monitoring tools and reports for network, storage, and inter-card communication. The pre-deployed network, storage, and inter-card communication fault monitoring tools are started to check the network connectivity of cluster nodes, storage read / write response speed, and inter-card data transmission success rate item by item. If the test results all meet the preset standards, it is confirmed that the distributed training cluster has the ability to carry out tasks and can execute distributed task training. If any test item does not meet the preset standards, the fault monitoring tools are used to locate the abnormal cluster node and the specific fault point, perform repair operations, and restart the testing until all test items meet the standards, thereby completing the pre-training test of the distributed training cluster. At the same time, the fault monitoring tool can generate specific instructions through the fault detection model to call and troubleshoot fault problems during the task training process.

[0086] In this embodiment, by pre-deploying and pre-training testing the distributed training cluster, it can be ensured that the distributed training cluster can perform task training normally after deployment, thereby avoiding interruption of the cluster due to hardware failure during training, preventing waste of computing resources and time, and thus improving the reliability and stability of the training process, and optimizing the cluster training performance and efficiency.

[0087] In one embodiment, after executing the corresponding processing operation based on the operable instructions in the processing scheme, the method further includes: real-time monitoring of the running status of the corresponding training task in the distributed training cluster, verifying the update status of the training logs in the shared storage and the integrity of the task breakpoint data, and determining whether the task has resumed normal operation; if the task has resumed normal operation, continuing to read the training logs in the shared storage at a preset cycle and inputting them into the fault detection big model to maintain the monitoring training process; if the task has not resumed normal operation, sending a manual troubleshooting notification to the operation and maintenance personnel through the deployment platform, the manual troubleshooting notification including key fault information, records of executed operations, and training logs and task breakpoint data in the shared storage; after completing the fault troubleshooting, triggering a restart command, calling the task breakpoint data in the shared storage, and re-issuing the training task to the distributed training cluster, so that the fault detection big model can resume the monitoring and inference process for the training logs.

[0088] For example, after the fault detection big data model detects a fault and generates a corresponding handling solution, it sends the handling solution to the scheduling backend, enabling the scheduling backend to reissue the training task to the distributed training cluster based on the handling solution. After the scheduling backend executes the corresponding processing operation based on the operable instructions in the handling solution, the fault detection big data model monitors the running status of the corresponding training task in the distributed training cluster in real time, and verifies the update status of the training log in the shared storage and the integrity of the task breakpoint data, thereby determining whether the task has resumed normal operation.

[0089] For example, if the task resumes normal operation, the fault detection model continues to read the training logs in the shared storage according to a preset cycle and performs real-time analysis on the training logs, thus maintaining the monitoring and training process. If the task fails to resume normal operation, the scheduling backend sends a manual troubleshooting notification to the operations and maintenance personnel through the deployment platform, including key fault information, records of executed operations, and training logs and task breakpoint data in the shared storage. After the operations and maintenance personnel complete the fault troubleshooting, the scheduling backend triggers a restart command, retrieves the task breakpoint data in the shared storage, and reissues the training task to the distributed training cluster, thereby enabling the fault detection model to resume its monitoring and inference process based on the training logs.

[0090] In this embodiment, after the scheduling backend executes the corresponding processing operation based on the operable instructions in the fault handling scheme, it reissues the training task to the distributed training cluster, so that the training task can resume normal operation, which can ensure the continuity of task training and the stability of cluster operation and maintenance, and improve the efficiency of task training.

[0091] like Figure 3 As shown, a specific embodiment illustrates a method for handling distributed training failures, including steps 302 to 310. Wherein,

[0092] Step 302: Pre-deploy the distributed training cluster and perform pre-training checks, while deploying the scheduling backend, the fault detection large model, and connecting to shared storage.

[0093] Specifically, the distributed cluster nodes are uniformly registered and managed through the Kubernetes container platform. Network communication protocols between nodes, storage connection interfaces, and inter-card data transmission parameters are configured to complete the pre-deployment of the distributed training cluster. The pre-deployed network, storage, and inter-card communication fault monitoring tools are then activated to test the network connectivity, storage read / write response speed, and inter-card data transmission success rate of the cluster nodes. Abnormal cluster nodes and specific fault points are located using the fault monitoring tools. For test items that do not meet the preset standards, repair operations are performed, and the testing is restarted. This ensures that each test item meets the preset standards, confirming that the distributed training cluster has the capacity to handle tasks, thus completing the pre-training check of the distributed training cluster.

[0094] Simultaneously deploy the scheduling backend and the fault detection big model, connect the fault detection big model to the scheduling backend and the distributed training cluster respectively, and connect the shared storage to the distributed training cluster and the fault detection big model respectively.

[0095] Step 304: The scheduling backend distributes distributed training tasks to the distributed training cluster and shares and stores the training logs, task breakpoint data and model-related information generated during the training process in real time.

[0096] Specifically, the scheduling backend distributes model training tasks, i.e. distributed training tasks, to the distributed training cluster. After the training task starts, the training logs, task breakpoints, and model-related information generated during the model training process are collected in real time and stored in shared storage.

[0097] Step 306: The fault detection big model reads the training logs in the shared storage according to a preset period, performs real-time analysis on the training logs, generates real-time information, and sends it to the scheduling backend.

[0098] Specifically, the fault detection big data model reads the training logs from the shared storage at preset intervals and performs real-time analysis on these logs. If the big data model determines that there is no abnormal information in the training logs, it sends the status information of normal task training to the scheduling backend, and generates model training progress, model validation accuracy, and training suggestions. The scheduling backend then feeds back the obtained training progress, model validation accuracy, and training suggestions to the deployment platform for users to view.

[0099] If the fault detection big model determines that there is abnormal information in the training log, it generates corresponding abnormal information and handling solutions based on the abnormality types detected by the fault detection big model, including node faults, business faults and other undefined faults. The obtained abnormal information is then fed back to the deployment platform through the scheduling backend for users to view.

[0100] Step 308: If there is abnormal information in the training log, the scheduling backend performs processing operations and resends the task based on the abnormal information and the processing plan.

[0101] Specifically, after the fault detection big model detects a fault and generates a corresponding solution, it sends the solution to the scheduling backend, which then reissues the training task to the distributed training cluster based on the solution until the task training is completed.

[0102] When the fault detection model detects a node failure, it extracts the node's abnormal features from the training logs, generates a special error code, and sends it to the scheduling backend. The scheduling backend locates the abnormal node based on the special error code and performs node isolation. At the same time, it calls the task breakpoint data in the shared storage to resend the training task to the distributed training cluster, thereby re-sending the task based on the breakpoint.

[0103] When the fault detection model detects a business-level fault, i.e., a code-level fault, it extracts the associated code location and path information from the training logs based on code-level faults, including coding errors and path errors. It then generates a processing plan containing code correction suggestions and path adjustment schemes, enabling the scheduling backend to execute the corresponding processing operations based on the actionable instructions in the plan. Simultaneously, the processing plan is fed back to the deployment platform for user viewing, and training logs, task breakpoint data, and model-related information are retained in shared storage for subsequent troubleshooting.

[0104] When the fault detection model detects an unknown fault without a clear solution, it summarizes the fault information, generates key information such as node operating parameters and fault triggering timing for this distributed task, and stores this information in shared storage. Simultaneously, it feeds back the acquired key information to the deployment platform to provide data support for manual troubleshooting. If a solution is provided through manual troubleshooting, the scheduling backend executes the corresponding processing operation based on the operable instructions in the solution. Furthermore, the fault detection model is iteratively trained using relevant training logs, task breakpoint data, model-related information, node operating parameters, and fault triggering timing from shared storage as input, and the manually provided solution as output. This continuously covers unknown faults, providing the fault detection model with multiple corresponding processing solutions for unknown faults.

[0105] Step 310: After the distributed training cluster attempts to restart the training task, the fault detection big model monitors the running status of the corresponding training task in the distributed training cluster in real time.

[0106] Specifically, after the scheduling backend redistributes the training task to the distributed training cluster based on the fault handling scheme, the fault detection big model monitors the running status of the corresponding training task in the distributed training cluster in real time, and verifies the update status of the training log in the shared storage and the integrity of the task breakpoint data, thereby determining whether the task has resumed normal operation.

[0107] If the task resumes normal operation, the fault detection big model continues to read the training logs in the shared storage according to the preset period and performs real-time analysis on the training logs, so that the fault detection big model can maintain the monitoring training process.

[0108] If the task fails to resume normal operation, the scheduling backend sends a manual troubleshooting notification to the operations and maintenance personnel through the deployment platform. This notification includes key fault information, records of executed operations, and training logs and task breakpoint data in shared storage. After the operations and maintenance personnel complete the fault troubleshooting, the scheduling backend triggers a restart command, retrieves the task breakpoint data in shared storage, and reissues the training task to the distributed training cluster. This enables the fault detection model to resume monitoring and inference processes based on the training logs.

[0109] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0110] Based on the same inventive concept, this application also provides a distributed training fault processing apparatus for implementing the distributed training fault processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more distributed training fault processing apparatus embodiments provided below can be found in the limitations of the distributed training fault processing method described above, and will not be repeated here.

[0111] In one exemplary embodiment, such as Figure 4As shown, a distributed training fault handling device 400 is provided, including: a distribution module 402, a judgment module 404, and a feedback module 406, wherein:

[0112] The distribution module 402 is used to distribute distributed training tasks to the distributed training cluster and to share and store training logs, task breakpoint data and model-related information generated during the training process in real time.

[0113] The judgment module 404 is used to read the training log in the shared storage according to a preset period and input the training log into the fault detection big model; if the fault detection big model determines that there is no abnormal information in the training log, it generates the model training progress, model verification accuracy and training suggestions; if the fault detection big model determines that there is abnormal information in the training log, it generates abnormal information and processing solutions according to the abnormality type.

[0114] The feedback module 406 is used to feed back the training progress, model validation accuracy, and training suggestions to the deployment platform for users to view when the model training progress, model validation accuracy, and training suggestions are obtained; when abnormal information and handling solutions are obtained, the abnormal information is fed back to the deployment platform for users to view, the corresponding handling operations are executed based on the operable instructions in the handling solution, and the abnormal information and handling solutions are associated with the task breakpoint data in shared storage to provide data support for abnormal handling and task restart.

[0115] In one embodiment, the distribution module is also used to configure and perform read / write operations on the shared storage, including: pre-configuring the bandwidth threshold and storage space threshold of the shared storage; configuring a process read / write coordination mechanism for the shared storage, and queuing and scheduling read / write requests from multiple processes for training logs, task breakpoint data and model-related information during distributed training through file locks or process scheduling priority allocation.

[0116] In one embodiment, the determination module is further configured to generate exception information and processing solutions according to the exception type, including: when a node failure is detected, extracting node exception features from the training logs, generating a dedicated error code and a processing solution for node isolation and task resending; locating the exception node based on the dedicated error code and performing isolation operations, while simultaneously calling the task breakpoint data in the shared storage and re-issuing the training task to the distributed training cluster to restart the task based on the breakpoint; when a business failure is detected, and the business failure is a code-level failure, generating a processing solution including code correction suggestions and path adjustment solutions, feeding the processing solution back to the deployment platform for user viewing, and retaining the training logs, task breakpoint data, and model-related information in the shared storage for subsequent investigation; wherein, code-level failures include code writing errors and path errors, and the fault detection model extracts the error-related code location and path information from the training logs; when an unknown failure without a clear solution is detected, sharing the node running parameters and the fault triggering timing to provide data support for manual investigation.

[0117] In one embodiment, the determination module is further configured to, in the event of an unknown fault without a clear solution, if a solution is provided through manual investigation, take the relevant training logs, task breakpoint data, model-related information, node running parameters, and fault triggering timing in the shared storage as inputs, and the manually provided solution as outputs, and iteratively train the fault detection large model.

[0118] In one embodiment, the distribution module is also used to execute the pre-deployment and pre-training testing process of the distributed training cluster, including: completing the unified registration and management of cluster nodes through the management container platform, configuring the network communication protocol between nodes, storage connection interface and inter-card data transmission parameters; starting the pre-deployed network, storage and inter-card communication fault monitoring tools to test the network connectivity of cluster nodes, storage read and write response speed and inter-card data transmission success rate item by item; if the test results all meet the preset standards, confirm that the distributed training cluster has the task carrying capacity; if there are test items that do not meet the preset standards, locate the abnormal cluster nodes and specific fault points through the fault monitoring tools, perform repair operations and restart the test until all test items meet the standards.

[0119] In one embodiment, the feedback module is further configured to: after executing the corresponding processing operation based on the operable instructions in the processing scheme, include: real-time monitoring of the running status of the corresponding training task in the distributed training cluster, verifying the update status of the training log in the shared storage and the integrity of the task breakpoint data, and determining whether the task has resumed normal operation; if the task has resumed normal operation, continue to read the training log in the shared storage according to a preset period and input it into the fault detection big model to maintain the monitoring training process; if the task has not resumed normal operation, send a manual troubleshooting notification to the operation and maintenance personnel through the deployment platform. The manual troubleshooting notification includes key fault information, records of executed operations, and training logs and task breakpoint data in the shared storage; after completing the fault troubleshooting, trigger a restart command, call the task breakpoint data in the shared storage, and reissue the training task to the distributed training cluster so that the fault detection big model can resume the monitoring and inference process for the training log.

[0120] Each module in the aforementioned distributed training fault handling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0121] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a distributed training fault handling method.

[0122] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0124] Distribute distributed training tasks to the distributed training cluster and share and store training logs, task breakpoint data and model-related information generated during the training process in real time.

[0125] The training logs in the shared storage are read according to a preset period and input into the fault detection big model. If the fault detection big model determines that there is no abnormal information in the training logs, the model training progress, model verification accuracy and training suggestions are generated. If the fault detection big model determines that there is abnormal information in the training logs, abnormal information and handling solutions are generated according to the abnormality type.

[0126] Once the model training progress, model validation accuracy, and training suggestions are obtained, they are fed back to the deployment platform for user viewing. Once anomaly information and handling solutions are obtained, the anomaly information is fed back to the deployment platform for user viewing. Based on the operable instructions in the handling solution, the corresponding handling operations are executed. The anomaly information and handling solutions are also associated with the task breakpoint data in shared storage to provide data support for anomaly handling and task restart.

[0127] In one embodiment, when the processor executes the computer program, it also performs the following steps: configuration and read / write process of shared storage, including: pre-configuring the bandwidth threshold and storage space threshold of shared storage; configuring a process read / write coordination mechanism for shared storage, and queuing and scheduling read / write requests from multiple processes for training logs, task breakpoint data and model-related information during distributed training by allocating file locks or process scheduling priorities.

[0128] In one embodiment, when the processor executes the computer program, it further performs the following steps: generating exception information and handling solutions according to the exception type, including: when a node failure is detected, extracting node exception features from the training log, generating a dedicated error code and a handling solution for node isolation and task resending; locating the abnormal node based on the dedicated error code and performing isolation operations, while calling the task breakpoint data in the shared storage, and re-issuing the training task to the distributed training cluster to restart the task based on the breakpoint; when a business failure is detected, and the business failure is a code-level failure, generating a handling solution including code correction suggestions and path adjustment solutions, feeding the handling solution back to the deployment platform for users to view, and retaining the training log, task breakpoint data, and model-related information in the shared storage for subsequent investigation; wherein, code-level failures include code writing errors and path errors, and the fault detection model extracts the error-related code location and path information from the training log; when an unknown failure without a clear solution is detected, sharing the node running parameters and the fault triggering time to provide data support for manual investigation.

[0129] In one embodiment, when the processor executes the computer program, it also performs the following steps: in the case of detecting an unknown fault without a clear solution, if a solution is provided through manual investigation, the processor takes the relevant training logs, task breakpoint data, model-related information, node running parameters and fault triggering timing in the shared storage as inputs and the manually provided solution as outputs to iteratively train the fault detection large model.

[0130] In one embodiment, when the processor executes the computer program, it also performs the following steps: a pre-deployment and pre-training testing process for the distributed training cluster, including: completing the unified registration and management of cluster nodes through a management container platform, configuring the network communication protocol between nodes, storage connection interface, and inter-card data transmission parameters; starting the pre-deployed network, storage, and inter-card communication fault monitoring tools to test the network connectivity, storage read / write response speed, and inter-card data transmission success rate of the cluster nodes item by item; if the test results all meet the preset standards, confirming that the distributed training cluster has the ability to carry out tasks; if any test item does not meet the preset standards, locating the abnormal cluster node and specific fault point through the fault monitoring tool, performing repair operations, and restarting the test until all test items meet the standards.

[0131] In one embodiment, when the processor executes the computer program, it further implements the following steps: after executing the corresponding processing operation based on the operable instructions in the processing scheme, it further includes: real-time monitoring of the running status of the corresponding training task in the distributed training cluster, and verifying the update status of the training log in the shared storage and the integrity of the task breakpoint data to determine whether the task has resumed normal operation; if the task has resumed normal operation, it continues to read the training log in the shared storage at a preset cycle and input it into the fault detection big model to maintain the monitoring training process; if the task has not resumed normal operation, it sends a manual troubleshooting notification to the operation and maintenance personnel through the deployment platform. The manual troubleshooting notification includes key fault information, records of executed operations, and training logs and task breakpoint data in the shared storage; after completing the fault troubleshooting, it triggers a restart command, calls the task breakpoint data in the shared storage, and reissues the training task to the distributed training cluster so that the fault detection big model can resume the monitoring and inference process for the training log.

[0132] The implementation principle and technical effects of the above embodiments are similar to those of the above method embodiments, and will not be repeated here.

[0133] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0134] Distribute distributed training tasks to the distributed training cluster and share and store training logs, task breakpoint data and model-related information generated during the training process in real time.

[0135] The training logs in the shared storage are read according to a preset period and input into the fault detection big model. If the fault detection big model determines that there is no abnormal information in the training logs, the model training progress, model verification accuracy and training suggestions are generated. If the fault detection big model determines that there is abnormal information in the training logs, abnormal information and handling solutions are generated according to the abnormality type.

[0136] Once the model training progress, model validation accuracy, and training suggestions are obtained, they are fed back to the deployment platform for user viewing. Once anomaly information and handling solutions are obtained, the anomaly information is fed back to the deployment platform for user viewing. Based on the operable instructions in the handling solution, the corresponding handling operations are executed. The anomaly information and handling solutions are also associated with the task breakpoint data in shared storage to provide data support for anomaly handling and task restart.

[0137] In one embodiment, when the computer program is executed by the processor, it also implements the following steps: configuration and read / write process of shared storage, including: pre-configuring the bandwidth threshold and storage space threshold of shared storage; configuring a process read / write coordination mechanism for shared storage, and queuing and scheduling read / write requests from multiple processes for training logs, task breakpoint data and model-related information during distributed training through file locks or process scheduling priority allocation.

[0138] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: generating exception information and handling solutions according to the exception type, including: when a node failure is detected, extracting node exception features from the training log, generating a dedicated error code and a handling solution for node isolation and task resending; locating the abnormal node based on the dedicated error code and performing isolation operations, while calling the task breakpoint data in shared storage, and re-issuing the training task to the distributed training cluster to restart the task based on the breakpoint; when a business failure is detected, and the business failure is a code-level failure, generating a handling solution including code correction suggestions and path adjustment solutions, feeding the handling solution back to the deployment platform for users to view, and retaining the training log, task breakpoint data, and model-related information in shared storage for subsequent investigation; wherein, code-level failures include code writing errors and path errors, and the fault detection model extracts the error-related code location and path information from the training log; when an unknown failure without a clear solution is detected, sharing the node running parameters and the fault triggering time to provide data support for manual investigation.

[0139] In one embodiment, when the computer program is executed by the processor, it also performs the following steps: in the case of detecting an unknown fault without a clear solution, if a solution is provided through manual investigation, the relevant training logs, task breakpoint data, model-related information, node running parameters and fault triggering timing in the shared storage are taken as input, and the manually provided solution is taken as output, and the fault detection large model is iteratively trained.

[0140] In one embodiment, when the computer program is executed by the processor, it also performs the following steps: a pre-deployment and pre-training testing process for the distributed training cluster, including: completing the unified registration and management of cluster nodes through a management container platform, configuring the network communication protocol between nodes, storage connection interface, and inter-card data transmission parameters; starting the pre-deployed network, storage, and inter-card communication fault monitoring tools to test the network connectivity, storage read / write response speed, and inter-card data transmission success rate of the cluster nodes item by item; if the test results all meet the preset standards, confirming that the distributed training cluster has the ability to carry out tasks; if any test item does not meet the preset standards, locating the abnormal cluster node and specific fault point through the fault monitoring tool, performing repair operations, and restarting the test until all test items meet the standards.

[0141] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: after executing the corresponding processing operation based on the operable instructions in the processing scheme, it further includes: real-time monitoring of the running status of the corresponding training task in the distributed training cluster, verifying the update status of the training log in the shared storage and the integrity of the task breakpoint data, and determining whether the task has resumed normal operation; if the task has resumed normal operation, continuing to read the training log in the shared storage at a preset cycle and inputting it into the fault detection big model to maintain the monitoring training process; if the task has not resumed normal operation, sending a manual troubleshooting notification to the operation and maintenance personnel through the deployment platform, the manual troubleshooting notification including key fault information, records of executed operations, and training logs and task breakpoint data in the shared storage; after completing the fault troubleshooting, triggering a restart command, calling the task breakpoint data in the shared storage, and re-issuing the training task to the distributed training cluster, so that the fault detection big model can resume the monitoring and inference process for the training log.

[0142] The implementation principle and technical effects of the above embodiments are similar to those of the above method embodiments, and will not be repeated here.

[0143] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0144] Distribute distributed training tasks to the distributed training cluster and share and store training logs, task breakpoint data and model-related information generated during the training process in real time.

[0145] The training logs in the shared storage are read according to a preset period and input into the fault detection big model. If the fault detection big model determines that there is no abnormal information in the training logs, the model training progress, model verification accuracy and training suggestions are generated. If the fault detection big model determines that there is abnormal information in the training logs, abnormal information and handling solutions are generated according to the abnormality type.

[0146] Once the model training progress, model validation accuracy, and training suggestions are obtained, they are fed back to the deployment platform for user viewing. Once anomaly information and handling solutions are obtained, the anomaly information is fed back to the deployment platform for user viewing. Based on the operable instructions in the handling solution, the corresponding handling operations are executed. The anomaly information and handling solutions are also associated with the task breakpoint data in shared storage to provide data support for anomaly handling and task restart.

[0147] In one embodiment, when the computer program is executed by the processor, it also implements the following steps: configuration and read / write process of shared storage, including: pre-configuring the bandwidth threshold and storage space threshold of shared storage; configuring a process read / write coordination mechanism for shared storage, and queuing and scheduling read / write requests from multiple processes for training logs, task breakpoint data and model-related information during distributed training through file locks or process scheduling priority allocation.

[0148] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: generating exception information and handling solutions according to the exception type, including: when a node failure is detected, extracting node exception features from the training log, generating a dedicated error code and a handling solution for node isolation and task resending; locating the abnormal node based on the dedicated error code and performing isolation operations, while calling the task breakpoint data in shared storage, and re-issuing the training task to the distributed training cluster to restart the task based on the breakpoint; when a business failure is detected, and the business failure is a code-level failure, generating a handling solution including code correction suggestions and path adjustment solutions, feeding the handling solution back to the deployment platform for users to view, and retaining the training log, task breakpoint data, and model-related information in shared storage for subsequent investigation; wherein, code-level failures include code writing errors and path errors, and the fault detection model extracts the error-related code location and path information from the training log; when an unknown failure without a clear solution is detected, sharing the node running parameters and the fault triggering time to provide data support for manual investigation.

[0149] In one embodiment, when the computer program is executed by the processor, it also performs the following steps: in the case of detecting an unknown fault without a clear solution, if a solution is provided through manual investigation, the relevant training logs, task breakpoint data, model-related information, node running parameters and fault triggering timing in the shared storage are taken as input, and the manually provided solution is taken as output, and the fault detection large model is iteratively trained.

[0150] In one embodiment, when the computer program is executed by the processor, it also performs the following steps: a pre-deployment and pre-training testing process for the distributed training cluster, including: completing the unified registration and management of cluster nodes through a management container platform, configuring the network communication protocol between nodes, storage connection interface, and inter-card data transmission parameters; starting the pre-deployed network, storage, and inter-card communication fault monitoring tools to test the network connectivity, storage read / write response speed, and inter-card data transmission success rate of the cluster nodes item by item; if the test results all meet the preset standards, confirming that the distributed training cluster has the ability to carry out tasks; if any test item does not meet the preset standards, locating the abnormal cluster node and specific fault point through the fault monitoring tool, performing repair operations, and restarting the test until all test items meet the standards.

[0151] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: after executing the corresponding processing operation based on the operable instructions in the processing scheme, it further includes: real-time monitoring of the running status of the corresponding training task in the distributed training cluster, verifying the update status of the training log in the shared storage and the integrity of the task breakpoint data, and determining whether the task has resumed normal operation; if the task has resumed normal operation, continuing to read the training log in the shared storage at a preset cycle and inputting it into the fault detection big model to maintain the monitoring training process; if the task has not resumed normal operation, sending a manual troubleshooting notification to the operation and maintenance personnel through the deployment platform, the manual troubleshooting notification including key fault information, records of executed operations, and training logs and task breakpoint data in the shared storage; after completing the fault troubleshooting, triggering a restart command, calling the task breakpoint data in the shared storage, and re-issuing the training task to the distributed training cluster, so that the fault detection big model can resume the monitoring and inference process for the training log.

[0152] The implementation principle and technical effects of the above embodiments are similar to those of the above method embodiments, and will not be repeated here.

[0153] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0155] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0156] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for handling distributed training faults, characterized in that, The method includes: Distribute distributed training tasks to the distributed training cluster and share and store training logs, task breakpoint data and model-related information generated during the training process in real time. The training logs in the shared storage are read according to a preset period and input into the fault detection big model. If the fault detection big model determines that there is no abnormal information in the training logs, the model training progress, model verification accuracy and training suggestions are generated. If the fault detection big model determines that there is abnormal information in the training logs, abnormal information and processing solutions are generated according to the abnormality type. Upon obtaining model training progress, model validation accuracy, and training suggestions, the system feeds these information back to the deployment platform for user review. Upon obtaining anomaly information and a handling plan, the system feeds this information back to the deployment platform for user review, executes corresponding processing operations based on the operable instructions in the handling plan, and associates the anomaly information and handling plan with task breakpoint data in shared storage to provide data support for anomaly handling and task restart.

2. The method according to claim 1, characterized in that, The configuration and read / write process of the shared storage includes: Pre-configure the bandwidth and storage space thresholds for shared storage; Configure a process read / write coordination mechanism for the shared storage, and queue and schedule read / write requests from multiple processes for training logs, task breakpoint data and model-related information during distributed training by allocating file locks or process scheduling priorities.

3. The method according to claim 1, characterized in that, The generation of exception information and processing solutions according to exception type includes: In the event of a node failure, extract the node's abnormal characteristics from the training logs, generate a dedicated error code and a handling scheme for node isolation and task resending; locate the abnormal node based on the dedicated error code and perform isolation operations, and at the same time call the task breakpoint data in the shared storage to resend the training task to the distributed training cluster to restart the task based on the breakpoint. Upon detecting a business failure, and if the failure is at the code level, a processing solution is generated that includes code correction suggestions and path adjustment plans. The processing solution is then fed back to the deployment platform for users to view, and training logs, task breakpoint data, and model-related information are retained in shared storage for subsequent troubleshooting. The code-level failure includes code writing errors and path errors. The fault detection model extracts the code location and path information associated with the errors from the training logs. In the event of an unknown fault with no clear solution, node operating parameters and fault triggering timing are shared and stored to provide data support for manual troubleshooting.

4. The method according to claim 3, characterized in that, The method further includes: In the event of an unknown fault with no clear solution, if a solution is provided through manual investigation, the relevant training logs, task breakpoint data, model-related information, node running parameters, and fault triggering timing in the shared storage are used as inputs, and the manually provided solution is used as output to iteratively train the fault detection model.

5. The method according to claim 1, characterized in that, The pre-deployment and pre-training testing process of the distributed training cluster includes: The management container platform enables unified registration and management of cluster nodes, and configures network communication protocols, storage connection interfaces, and data transfer parameters between cards. Activate the pre-deployed network, storage, and inter-card communication fault monitoring tools to test the network connectivity of cluster nodes, storage read / write response speed, and inter-card data transmission success rate item by item. If all test results meet the preset standards, the distributed training cluster is confirmed to have the ability to carry out tasks. If any test item does not meet the preset standards, the abnormal cluster node and the specific fault point are located by the fault monitoring tool, the repair operation is performed and the test is restarted until all test items meet the standards.

6. The method according to claim 1, characterized in that, After executing the corresponding processing operation based on the operable instructions in the processing scheme, the method further includes: Real-time monitoring of the running status of the corresponding training tasks in the distributed training cluster, verification of the update status of training logs in shared storage and the integrity of task breakpoint data, and determination of whether the task has resumed normal operation. If the task resumes normal operation, continue to read the training logs in the shared storage at the preset cycle and input them into the fault detection big model to maintain the monitoring training process; If the task fails to return to normal operation, a manual troubleshooting notification is sent to the operations and maintenance personnel through the deployment platform. The manual troubleshooting notification includes key fault information, records of executed operations, and training logs and task breakpoint data in shared storage. After the fault troubleshooting is completed, a restart command is triggered to call the task breakpoint data in shared storage and reissue the training task to the distributed training cluster so that the fault detection model can resume the monitoring and inference process for training logs.

7. A distributed training fault handling device, characterized in that, The device includes: The distribution module is used to distribute training tasks to the distributed training cluster and to share and store training logs, task breakpoint data and model-related information generated during the training process in real time. The judgment module is used to read the training logs in the shared storage according to a preset period, and input the training logs into the fault detection big model; if the fault detection big model determines that there is no abnormal information in the training logs, it generates model training progress, model verification accuracy and training suggestions; if the fault detection big model determines that there is abnormal information in the training logs, it generates abnormal information and processing solutions according to the abnormality type. The feedback module is used to feed back the training progress, model validation accuracy, and training suggestions to the deployment platform for user viewing when model training progress, model validation accuracy, and training suggestions are obtained; when abnormal information and handling solutions are obtained, the abnormal information is fed back to the deployment platform for user viewing, the corresponding processing operation is executed based on the operable instructions in the handling solution, and the abnormal information and handling solutions are associated with task breakpoint data in shared storage to provide data support for abnormal handling and task restart.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images