Fault repair method and device for model training, equipment and storage medium
By deploying computing management resources and fault management resources in Kubernetes training clusters, and automatically performing model training and fault repair, the problem of frequent failures in model training is solved, and training efficiency and monitoring recovery efficiency are improved.
Patent Information
- Application Number
- CN202411680868.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-03-11
AI Technical Summary
During the model training process, failures occur frequently, resulting in an increase in training time, low monitoring and recovery efficiency, affecting the model training efficiency.
By obtaining the user's compute resource type, create a Kubernetes training cluster, and deploy compute management resources and fault management resources in the cluster, including monitoring services, fault detection services, and fault repair services. Use these resources to automatically conduct model training, fault detection and fault repair to reduce human intervention.
Automatic model training and fault repair are realized, monitoring and recovery efficiency is improved, human intervention is reduced, and model training is improved.
Smart Images
Figure CN119597524A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a fault repair method, device, equipment and storage medium for model training. Background Art
[0002] Nowadays, model training, such as large-scale pre-trained models in deep learning, such as BERT, GPT-3, DALL·E, etc., is an important progress in the field of artificial intelligence in recent years. As the model becomes larger and the amount of data increases, the training time of model training becomes longer and longer, resulting in more and more failures in the model pre-training process. For example, Meta's 22-year training 175B log. The two-week log shows that there were 40 interruptions in training for various reasons, including hardware, infrastructure, loss divergence, etc. The longest continuous training time in two weeks was only 2.8 days, 2 days and 1.5 days, which means that there were interruptions almost every day. Therefore, in order to reduce the impact of failures, some companies directly deploy pre-training programs such as PyTorch and TensorFlow on bare metal, gpu cloud host or docker, and then monitor the machine operation through monitoring programs. It is manually determined whether there is an abnormality based on the monitored operation, which requires a lot of human intervention, and the monitoring and recovery efficiency is low, which reduces the efficiency of model training. Summary of the invention
[0003] The embodiments of the present application provide a fault repair method, device, equipment and storage medium for model training to solve at least one problem existing in the related art. The technical solution is as follows:
[0004] In a first aspect, an embodiment of the present application provides a method for repairing a fault in model training, comprising:
[0005] Get the user's computing resource type;
[0006] Create a Kubernetes training cluster, and deploy corresponding service resources in the Kubernetes training cluster according to the computing resource type, wherein the service resources include computing management resources and fault management resources;
[0007] The to-be-trained model is trained by the computing management resource, and during the training process, fault detection is performed by the fault management resource to obtain a fault detection result;
[0008] When the fault detection result indicates that a fault exists, the fault is repaired by using the fault management resource.
[0009] In one embodiment, deploying corresponding service resources in the Kubernetes training cluster according to the computing resource type includes:
[0010] In the Kubernetes training cluster, deploy the monitoring service Prometheus, the fault detection service, and the fault repair service to obtain fault management resources;
[0011] In the Kubernetes training cluster, deploy the Volcano scheduler and the distributed data management plug-in fluid, and when the computing resource type is GPU, install the NVIDIA GPU plug-in nvidia-plugin in the Kubernetes training cluster, and when the computing resource type is NPU, install the AscendNPU plug-in ascend-plugin in the Kubernetes training cluster;
[0012] According to the Volcano scheduler and the distributed data management plug-in fluid, and the NVIDIA GPU plug-in nvidia-plugin or the Ascend NPU plug-in ascend-plugin, computing management resources are obtained.
[0013] In one implementation, the training of the to-be-trained model by the computing management resource includes:
[0014] Use the Volcano scheduler to perform batch computing and training task scheduling for each node in the Kubernetes training cluster.
[0015] Through the NVIDIA GPU plug-in nvidia-plugin or the Ascend NPU plug-in ascend-plugin, several nodes of the Kubernetes training cluster use GPU or NPU to train the model to be trained;
[0016] Distributed data management of the training data of each of the nodes is performed through the distributed data management plug-in fluid.
[0017] In one implementation, during the training process, performing fault detection by using the fault management resource to obtain a fault detection result includes:
[0018] During the training process, status information is obtained through the monitoring service Prometheus; the status information includes Pod status, node status, network status and computing resource status, and the computing resource status is GPU status or NPU status;
[0019] Through the fault detection service, fault detection is performed on the state information to obtain a fault detection result.
[0020] In one implementation, when the fault detection result indicates that a fault exists, repairing the fault by using the fault management resource includes:
[0021] When the fault detection result indicates that a fault exists:
[0022] If the fault is an abnormal GPU state, reduce the GPU load, restart the GPU, or switch to a standby GPU through the fault recovery service;
[0023] If the fault is that the NPU state is abnormal, reduce the GPU load, restart the GPU, or switch to a standby NPU through the fault recovery service;
[0024] If the fault is an abnormal network status, the network is reconfigured or a backup network is switched through the fault recovery service.
[0025] In one implementation, when the fault detection result indicates that a fault exists, repairing the fault by using the fault management resource includes:
[0026] When the fault detection result indicates that a fault exists:
[0027] If the fault is an abnormal Pod status, restart the Pod through the fault repair service, and if the Pod status is still abnormal after the Pod is restarted, migrate the abnormal Pod to other nodes;
[0028] If the fault is an abnormal node status, the abnormal node with abnormal node status is repaired by the fault repair service, and when the abnormal node repair fails, a spare node is added to the Kubernetes training cluster, and the spare node replaces the abnormal node to continue training.
[0029] In one implementation, replacing the abnormal node with the standby node and continuing the training includes:
[0030] Saving the training data of the abnormal node to hot storage;
[0031] The label of the abnormal node is marked on the standby node to reconfigure the standby node, and the training data in the hot storage is loaded through the standby node to continue training.
[0032] In a second aspect, an embodiment of the present application provides a fault repair device for model training, comprising:
[0033] An acquisition module is used to obtain the user's computing resource type;
[0034] A deployment module, used to create a Kubernetes training cluster and deploy corresponding service resources in the Kubernetes training cluster according to the computing resource type, wherein the service resources include computing management resources and fault management resources;
[0035] A detection module, used to train the to-be-trained model through the computing management resource, and to perform fault detection through the fault management resource during the training process to obtain a fault detection result;
[0036] The repair module is used to repair the fault by using the fault management resource when the fault detection result indicates that a fault exists.
[0037] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement a method in any one of the above-mentioned embodiments.
[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, the method in any one of the above-mentioned embodiments is implemented.
[0039] The beneficial effects of the above technical solution include at least:
[0040] By obtaining the user's computing resource type, a Kubernetes training cluster is created, and according to the computing resource type, corresponding service resources are deployed in the Kubernetes training cluster, the to-be-trained model is trained through the computing management resources, and during the training process, fault detection is performed through the fault management resources to obtain a fault detection result. When the fault detection result indicates the existence of a fault, the fault is repaired through the fault management resources, and the Kubernetes training cluster is introduced to deploy computing management resources and service resources such as fault management resources, so as to automatically perform model training, fault detection and fault repair, reduce human intervention, improve the efficiency of monitoring and recovery, and thus improve the efficiency of model training.
[0041] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.
[0043] Figure 1 A schematic diagram of the steps of a fault repair method for model training according to an embodiment of the present application;
[0044] Figure 2 A structural block diagram of a fault repair device for model training according to an embodiment of the present application;
[0045] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0046] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.
[0047] Reference Figure 1 , a flow chart of a fault repair method for model training according to an embodiment of the present application is shown, and the fault repair method for model training may at least include steps S100-S400:
[0048] S100: Obtain the computing resource type of the user.
[0049] S200. Create a Kubernetes training cluster and deploy corresponding service resources in the Kubernetes training cluster according to the computing resource type. The service resources include computing management resources and fault management resources.
[0050] S300: Train the to-be-trained model by using computing management resources, and perform fault detection by using fault management resources during the training process to obtain a fault detection result.
[0051] S400: When the fault detection result indicates that a fault exists, the fault is repaired by using fault management resources.
[0052] The fault repair method of model training in the embodiment of the present application can be executed through a computer, mobile phone, tablet or server, for example, on a server of an intelligent computing platform (referred to as the platform).
[0053] The technical solution of the embodiment of the present application obtains the computing resource type of the user, creates a Kubernetes training cluster, and deploys corresponding service resources in the Kubernetes training cluster according to the computing resource type, trains the to-be-trained model through computing management resources, and performs fault detection through fault management resources during the training process to obtain fault detection results. When the fault detection result indicates the existence of a fault, the fault is repaired through fault management resources, and computing management resources and other service resources such as fault management resources are deployed in the Kubernetes training cluster to automatically perform model training, fault detection, and fault repair, thereby reducing human intervention, improving the efficiency of monitoring and recovery, and thus improving the efficiency of model training.
[0054] In one implementation, the user can determine the type of computing resources that can be used according to his or her own needs, such as a GPU or an NPU, and the platform obtains the computing resource type of the user.
[0055] In one implementation, the administrator installs a Kubernetes training cluster on the platform in advance, for example, the process is:
[0056] #Install Kubernetes
[0057] curl -s https: / / packages.cloud.google.com / apt / doc / apt-key.gpg|apt-keyadd-
[0058] echo"deb http: / / apt.kubernetes.io / kubernetes-xenial main"> / etc / apt / sources.list.d / kubernetes.list
[0059] apt-get update
[0060] apt-get install-y kubelet kubeadm kubectl
[0061] #Initialize Kubernetes cluster
[0062] kubeadminit --pod-network-cidr=10.244.0.0 / 16
[0063] #Configure the kubectl command line tool
[0064] mkdir -p $HOME / .kube
[0065] cp-i / etc / kubernetes / admin.conf$HOME / .kube / config
[0066] chown$(id-u):$(id-g)$HOME / .kube / config
[0067] #Install the Flannel network plug-in
[0068] kubectl apply-f kube-flannel.yml
[0069] Wherein, in step S200, corresponding service resources are deployed in the Kubernetes training cluster according to the computing resource type, including steps S210-S230:
[0070] S210. In the Kubernetes training cluster, deploy the monitoring service Prometheus, the fault detection service, and the fault repair service to obtain fault management resources.
[0071] Optionally, service resources include computing management resources and fault management resources, and management resources may include services and plug-ins. In an embodiment of the present application, a monitoring service Prometheus (for platform and service monitoring), a fault detection service, and a fault repair service (for example, the repair of a GPU corresponds to auto-fix-faults) are deployed in a Kubernetes training cluster to obtain fault management resources. It should be noted that if the computing resource type is GPU, the fault detection service corresponds to gpu-error-detect, and other services or plug-ins for detecting Pods, node status, etc. are also included, without specific limitation; wherein, when the fault detection service is deployed, it will automatically run once to predict abnormal problems in the current cluster in advance.
[0072] S220. In the Kubernetes training cluster, deploy the Volcano scheduler and the distributed data management plug-in fluid. When the computing resource type is GPU, install the NVIDIA GPU plug-in nvidia-plugin in the Kubernetes training cluster. When the computing resource type is NPU, install the Ascend NPU plug-in ascend-plugin in the Kubernetes training cluster.
[0073] S230 , obtaining computing management resources according to the Volcano scheduler, the distributed data management plug-in fluid, and the NVIDIA GPU plug-in nvidia-plugin or the Ascend NPU plug-in ascend-plugin.
[0074] Optionally, in the Kubernetes training cluster, deploy the Volcano scheduler (for batch computing and task scheduling) and the distributed data management plug-in fluid (for distributed data management), and when the computing resource type is GPU, install the NVIDIA GPU plug-in nvidia-plugin (for managing NVIDIA GPU resources) in the Kubernetes training cluster, and when the computing resource type is NPU, install the Ascend NPU plug-in ascend-plugin (for managing Ascend NPU resources) in the Kubernetes training cluster. Therefore, based on the Volcano scheduler and the distributed data management plug-in fluid, as well as the NVIDIA GPU plug-in nvidia-plugin or the Ascend NPU plug-in ascend-plugin, computing management resources are finally obtained.
[0075] In one implementation, step S300 trains the model to be trained by computing management resources, including steps S310-S330:
[0076] S310, through the Volcano scheduler, batch computing and training task scheduling of each node in the Kubernetes training cluster.
[0077] Optionally, when using the Kubernetes training cluster to train the model to be trained, in order to ensure the efficiency and effectiveness of the training, the Volcano scheduler is used to perform batch calculations and training task scheduling for each node of the Kubernetes training cluster, reasonably determine the number of nodes and the computing resources allocated to each node, and ensure the efficient operation of the training task.
[0078] S320, through the NVIDIA GPU plug-in nvidia-plugin or Ascend NPU plug-in ascend-plugin, enable several nodes of the Kubernetes training cluster to use GPU or NPU to train the model to be trained.
[0079] Optionally, several nodes can be determined according to the Volcano scheduler. When the computing resource type is GPU, the NVIDIA GPU plug-in nvidia-plugin is used to enable several nodes of the Kubernetes training cluster to use GPU to train the model to be trained; when the computing resource type is NPU, the Ascend NPU plug-in ascend-plugin is used to enable several nodes of the Kubernetes training cluster to use NPU to train the model to be trained.
[0080] S330. Distributed data management of training data of each node is performed through the distributed data management plug-in fluid.
[0081] Optionally, since model training requires the use of training data, the distributed data management plug-in fluid is used to perform distributed data management of the training data of each node, such as how to allocate training data, update training data, etc., to ensure the smooth progress of the training process.
[0082] In one implementation, during the training process in step S300, fault detection is performed by using the fault management resource to obtain a fault detection result, including steps S340-S350:
[0083] S340. During the training process, status information is obtained through the monitoring service Prometheus.
[0084] Optionally, during the training process, the monitoring service Prometheus monitors in real time to obtain status information, including but not limited to Pod status (Pod startup, running, failure, etc.), node status (Node resource usage, running status, etc.), network status, and computing resource status. The computing resource status is GPU status (such as hardware status, driver status, temperature, power consumption, utilization, single-machine single-card collective communication detection status, multi-machine multi-card collective communication detection status, etc.) or NPU status, and the corresponding computing resource type is GPU or NPU.
[0085] S350: Perform fault detection on the status information through a fault detection service to obtain a fault detection result.
[0086] Optionally, through the fault detection service, real-time fault detection is performed on the status information to obtain fault detection results to determine whether there are any abnormalities in the Pod status, node status, network status (such as the status of the ROCE network and the status of the InfiniBand network), and computing resource status, so as to predict potential problems in the cluster in advance and provide basic data for subsequent fault handling.
[0087] In one implementation, in step S400, when the fault detection result indicates that a fault exists, at least one of steps S410-S450 is included:
[0088] S410: If the fault is that the GPU state is abnormal, reduce the GPU load, restart the GPU, or switch to a standby GPU through a fault recovery service.
[0089] Optionally, if the fault is an abnormal GPU state, the platform's fault repair service can reduce the GPU load or restart the GPU. If reducing the GPU load or restarting the GPU is still abnormal, the fault repair service can switch to a backup GPU to ensure the normal progress of the training process. In some embodiments, if none of the above measures can solve the problem, a command can be sent based on the error code of the GPU state abnormality and the preset repair method provided by the GPU manufacturer to automatically repair the GPU card. After the repair, a test is performed. If the repair is not possible, the node with the abnormal GPU state is isolated and added as a backup node; or, for nodes that cannot be automatically repaired, a Prometheus alarm is issued, and human intervention is performed to repair and replace.
[0090] S420: If the fault is that the NPU state is abnormal, reduce the GPU load, restart the GPU, or switch to a standby NPU through a fault recovery service.
[0091] Similarly, similar to the GPU status abnormality, if the fault is the NPU status abnormality, the fault recovery service can reduce the GPU load, restart the GPU, or switch to one of the standby NPUs.
[0092] S430: If the fault is an abnormal network status, reconfigure the network or switch to a backup network through a fault recovery service.
[0093] Optionally, if the fault is caused by an abnormal network status, the fault recovery service will reconfigure the network. If the network status is still abnormal after reconfiguring the network, the fault recovery service can switch to a backup network to ensure the normal progress of the training process.
[0094] S440. If the fault is an abnormal Pod status, restart the Pod through the fault repair service. If the Pod status is still abnormal after the Pod is restarted, migrate the abnormal Pod to other nodes.
[0095] Optionally, if the fault is an abnormal Pod state, the Pod is restarted through the fault repair service. If the Pod state is still abnormal after the Pod is restarted, the abnormal Pod is migrated to other nodes until the Pod state of the Pod is normal.
[0096] S450. If the fault is an abnormal node status, the abnormal node with abnormal node status is repaired through the fault repair service. When the abnormal node repair fails, a spare node is added to the Kubernetes training cluster, and the spare node replaces the abnormal node to continue training.
[0097] Optionally, if the fault is an abnormal node status, the abnormal node with an abnormal node status is repaired through the fault repair service, and the node status of the node is re-detected after the repair. If the node status of the node is still abnormal, it means that the abnormal node repair has failed. Among them, when the abnormal node repair fails, a new spare node is added to the Kubernetes training cluster, the training data of the abnormal node is saved to the hot storage, and the label of the abnormal node is marked on the spare node to reconfigure the spare node, so as to load the training data in the hot storage through the spare node, continue training, isolate the abnormal node, and ensure the normal progress of model training.
[0098] The embodiment of the present application introduces a kubernetes cluster and implements automatic fault detection, automatic fault isolation, and automatic repair to improve the efficiency of model training. When a node failure is found, training will be stopped, training data will be saved to hot storage, and an attempt will be made to automatically repair the faulty node. If the repair is not possible, the faulty node will be isolated, a spare node will be added, and then the saved data will be added to continue training to ensure the normal progress of model training.
[0099] The method of the embodiment of the present application realizes automatic configuration, automatically performs comprehensive detection of the GPU without manual intervention, and supports multiple computing resources: whether the user chooses GPU or NPU, it can be flexibly configured through the corresponding plug-ins (such as nvidia-plugin and ascend-plugin), with good flexibility; it is highly customized, and can customize monitoring indicators and detection items according to the specific needs of users to ensure that the needs of different application scenarios are met; extensible plug-ins and services, users can add or delete plug-ins and services as needed, such as volcano for batch computing scheduling, fluid for distributed data management, etc.; cross-platform support, compatible with multiple hardware and software environments: the method is applicable to various hardware platforms (such as NVIDIA GPU and Ascend NPU) and software environments (such as Kubernetes clusters), and has good cross-platform capabilities; support for multiple network environments: it can monitor and process ROCE and IB network status, and adapt to different network environments.
[0100] Reference Figure 2 , shows a structural block diagram of a fault repair device for model training according to an embodiment of the present application, and the device may include:
[0101] An acquisition module is used to obtain the user's computing resource type;
[0102] The deployment module is used to create a Kubernetes training cluster and deploy corresponding service resources in the Kubernetes training cluster according to the computing resource type. The service resources include computing management resources and fault management resources.
[0103] The detection module is used to train the to-be-trained model through computing management resources, and to perform fault detection through fault management resources during the training process to obtain fault detection results;
[0104] The repair module is used to repair the fault through fault management resources when the fault detection result indicates that a fault exists.
[0105] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and will not be repeated here.
[0106] Reference Figure 3 , shows a structural block diagram of an electronic device according to an embodiment of the present application, the electronic device includes: a memory 310 and a processor 320, the memory 310 stores instructions that can be run on the processor 320, and the processor 320 loads and executes the instructions to implement the fault repair method of model training in the above embodiment. The number of the memory 310 and the processor 320 can be one or more.
[0107] In one embodiment, the electronic device also includes a communication interface 330 for communicating with external devices and performing data exchange transmission. If the memory 310, the processor 320 and the communication interface 330 are implemented independently, the memory 310, the processor 320 and the communication interface 330 can be interconnected through a bus and complete mutual communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0108] Optionally, in a specific implementation, if the memory 310, the processor 320 and the communication interface 330 are integrated on a chip, the memory 310, the processor 320 and the communication interface 330 can communicate with each other through an internal interface.
[0109] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the fault repair method for model training provided in the above embodiment.
[0110] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.
[0111] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.
[0112] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the advanced RISC machines (ARM) architecture.
[0113] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory, and may also include a non-volatile random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct rambus RAM (DR RAM).
[0114] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.
[0115] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.
[0116] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0117] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.
[0118] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.
[0119] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.
[0120] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.
[0121] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A fault repair method for model training, characterized in that: include: Get the user's computing resource type; Create a Kubernetes training cluster, and deploy corresponding service resources in the Kubernetes training cluster according to the computing resource type, wherein the service resources include computing management resources and fault management resources; The to-be-trained model is trained by the computing management resource, and during the training process, fault detection is performed by the fault management resource to obtain a fault detection result; When the fault detection result indicates that a fault exists, the fault is repaired by using the fault management resource.
2. The fault repair method of model training according to claim 1, characterized in that: Deploying corresponding service resources in the Kubernetes training cluster according to the computing resource type includes: In the Kubernetes training cluster, deploy the monitoring service Prometheus, the fault detection service, and the fault repair service to obtain fault management resources; In the Kubernetes training cluster, a Volcano scheduler and a distributed data management plug-in fluid are deployed, and when the computing resource type is GPU, an NVIDIA GPU plug-in nvidia-plugin is installed in the Kubernetes training cluster, and when the computing resource type is NPU, an Ascend NPU plug-in ascend-plugin is installed in the Kubernetes training cluster; According to the Volcano scheduler and the distributed data management plug-in fluid, and the NVIDIA GPU plug-in nvidia-plugin or the Ascend NPU plug-in ascend-plugin, computing management resources are obtained.
3. The fault repair method of model training according to claim 2, characterized in that: The training of the to-be-trained model by the computing management resource includes: Use the Volcano scheduler to perform batch computing and training task scheduling for each node in the Kubernetes training cluster. Through the NVIDIA GPU plug-in nvidia-plugin or the Ascend NPU plug-in ascend-plugin, several nodes of the Kubernetes training cluster use GPU or NPU to train the model to be trained; Distributed data management of the training data of each of the nodes is performed through the distributed data management plug-in fluid.
4. The fault repair method of model training according to claim 2 or 3, characterized in that: During the training process, fault detection is performed by using the fault management resource to obtain a fault detection result including: During the training process, status information is obtained through the monitoring service Prometheus; the status information includes Pod status, node status, network status and computing resource status, and the computing resource status is GPU status or NPU status; Through the fault detection service, fault detection is performed on the state information to obtain a fault detection result.
5. The fault repair method of model training according to claim 4, characterized in that: When the fault detection result indicates that a fault exists, repairing the fault by using the fault management resource includes: When the fault detection result indicates that a fault exists: If the fault is an abnormal GPU state, reduce the GPU load, restart the GPU, or switch to a standby GPU through the fault recovery service; If the fault is that the NPU state is abnormal, reduce the GPU load, restart the GPU, or switch to a standby NPU through the fault recovery service; If the fault is an abnormal network status, the network is reconfigured or a backup network is switched through the fault recovery service.
6. The fault repair method of model training according to claim 4, characterized in that: When the fault detection result indicates that a fault exists, repairing the fault by using the fault management resource includes: When the fault detection result indicates that a fault exists: If the fault is an abnormal Pod status, restart the Pod through the fault repair service, and if the Pod status is still abnormal after the Pod is restarted, migrate the abnormal Pod to other nodes; If the fault is an abnormal node status, the abnormal node with abnormal node status is repaired by the fault repair service, and when the abnormal node repair fails, a spare node is added to the Kubernetes training cluster, and the spare node replaces the abnormal node to continue training.
7. The fault repair method of model training according to claim 6, characterized in that: The step of replacing the abnormal node with the standby node and continuing the training includes: Saving the training data of the abnormal node to hot storage; The label of the abnormal node is marked on the standby node to reconfigure the standby node, and the training data in the hot storage is loaded through the standby node to continue training.
8. A fault repair device for model training, characterized in that: include: An acquisition module is used to obtain the user's computing resource type; A deployment module, used to create a Kubernetes training cluster and deploy corresponding service resources in the Kubernetes training cluster according to the computing resource type, wherein the service resources include computing management resources and fault management resources; A detection module, used to train the to-be-trained model through the computing management resource, and to perform fault detection through the fault management resource during the training process to obtain a fault detection result; The repair module is used to repair the fault by using the fault management resource when the fault detection result indicates that a fault exists.
9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.