Node control method, system and device based on Kubernetes, medium and program product

Abstract

The invention relates to the technical field of computer networks, and discloses a Kubernetes-based node control method, system and device, a medium and a program product.The node control method comprises the steps that in the large model distributed training process, fault log information of nodes is obtained through a Kubernetes node problem detection module; the fault log information is used for representing the fault type of the node; in the large model distributed training process, corresponding training tasks are distributed on callable nodes of the GPU; updating the node state of the node through the API server based on the fault log information; the node state comprises whether the node can be called. The fault log information of the GPU is obtained through a Kubernetes node problem detection module, the node proceeding state is updated, the training task is rescheduled according to the node state, it is guaranteed that the training task is only scheduled to a healthy node, and the training efficiency of a large model is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer network technology, and in particular to a node control method, system, device, medium and program product based on Kubernetes. Background Art

[0002] Currently, implementing elastic distributed training for large models faces many challenges, such as discovery and coordination between training processes, member change management, and integration with existing code. The training of large models relies on GPUs, and the Kubernetes Apiserver (Api server) cannot detect node anomalies caused by GPU failures in a timely manner by default, so instances may be scheduled to unavailable nodes during scheduling. At the same time, the existing elastic training frameworks cannot detect underlying GPU device errors, resulting in the user side being unable to stop losses in a timely manner.

[0003] Therefore, there is an urgent need for a node control method based on Kubernetes that can perceive GPU faulty nodes and adjust the allocation of training tasks in a timely manner. Summary of the invention

[0004] In view of this, the present application provides a node control method, system, device, medium and program product based on Kubernetes, which can sense the faulty nodes of GPU and adjust the allocation of training tasks in time. The technical solution is as follows.

[0005] In a first aspect, the present invention provides a node control method based on Kubernetes, the method comprising:

[0006] During the large-model distributed training process, the fault log information of the node is obtained through the node problem detection module of Kubernetes; the fault log information is used to characterize the fault type of the node; during the large-model distributed training process, the corresponding training tasks are allocated to the callable nodes of the GPU;

[0007] Based on the fault log information, the node status of the node is updated through the API server of Kubernetes; the node status includes whether the node is callable.

[0008] The node control method based on Kubernetes provided by the present invention has the following advantages.

[0009] The node control method based on Kubernetes of the present invention is applied in the process of large model distributed training. Large model distributed training relies on GPU, and the training task is assigned to the node of GPU for distributed training. The method of the present invention detects the GPU node in the training process through the node problem detection module of Kubernetes, obtains the fault log information and sends it to the API server, the fault log information contains the fault type of the node, and the fault type includes temporary fault and permanent fault. The API server updates the state of the node by the node fault type indicated by the fault log information. If the node is a temporary fault, the state of the node is updated to be callable, and the training task can continue to be assigned to the node; if the node is a permanent fault, the state of the node is updated to be uncallable. After the node state becomes uncallable, the node will be stained, unless the training task can tolerate the stain, otherwise the training task will no longer be assigned to the node. The fault log information not only includes the fault type of the node, but also includes whether the training task is interrupted. If the training task is interrupted due to a GPU device failure, the current training task is deleted. The node problem detection module of Kubernetes is used to obtain GPU fault log information, identify various GPU faults, and handle different faults in different ways. The status of the GPU node is updated according to the fault log information, and the training tasks are rescheduled according to the node status. This ensures that the training tasks of large model distributed training will only be scheduled to healthy nodes, thereby significantly improving the training efficiency of large models.

[0010] In an optional implementation, the node failure type includes a temporary node failure and a permanent node failure.

[0011] In an optional implementation, the Kubernetes-based node control method further includes: if the node is temporarily faulty, setting the node as callable through the Kubernetes API server; if the node is permanently faulty, setting the node as uncallable through the Kubernetes API server.

[0012] The node control method based on Kubernetes of the present invention determines whether the node needs to be shielded according to the node status of the GPU node, ensuring that the training task will only be assigned to the callable node, thereby ensuring the training efficiency of large-model distributed training.

[0013] In an optional embodiment, the Kubernetes-based node control method also includes: based on the node status of the node, allocating the first training task in the training tasks to an uncallable node; the uncallable node is set with a tolerance; the first training task is a training task in the training tasks that can meet the tolerance.

[0014] The Kubernetes-based node control method of the present invention, after analyzing the fault log information, adds a taint label to the uncallable node, and no longer assigns training tasks to the taint unless the training task can tolerate the taint, to ensure that certain specific training tasks can be executed on nodes (i.e., tainted nodes) with limited resources, poor performance, or other potential problems.

[0015] In an optional implementation, the fault log is also used to indicate whether the training task is interrupted; if the training task is interrupted, the training task is deleted through the Kubernetes API server.

[0016] The Kubernetes-based node control method of the present invention can also determine whether the training task is interrupted due to a GPU device failure by analyzing the fault log information, thereby determining whether to delete the current training task.

[0017] In a second aspect, the present invention provides a Kubernetes-based system, the system comprising a node problem detection module and an API server;

[0018] The node problem detection module is used to obtain the fault log information of the node during the large model distributed training process; the fault log information is used to characterize the fault type of the node; during the large model distributed training process, the corresponding training tasks are allocated to the callable nodes of the GPU;

[0019] The API server is used to update the node status of the node based on the fault log information; the node status includes whether the node is callable.

[0020] In an optional implementation, the node problem detection module is further used to: analyze whether the training task is interrupted according to the fault log information; if the training task is interrupted, delete the training task through the Kubernetes API server.

[0021] In an optional embodiment, the API server is also used to: assign the first training task among the training tasks to an uncallable node based on the node status of the node; the uncallable node is set with a tolerance; the first training task is a training task among the training tasks that can meet the tolerance.

[0022] In an optional embodiment, the Kubernetes-based system also includes: a task scheduling module, which is used to assign the first training task in the training tasks to an uncallable node based on the node status of the node; the uncallable node is set with a tolerance; the first training task is a training task in the training tasks that can meet the tolerance.

[0023] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the Kubernetes-based node control method of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the Kubernetes-based node control method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0025] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the Kubernetes-based node control method of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 It is a method flow chart of a node control method based on Kubernetes provided by an embodiment of the present invention.

[0028] Figure 2 It is a logic block diagram of a node control method based on Kubernetes according to an exemplary embodiment.

[0029] Figure 3 It is a structural diagram of a node control device based on Kubernetes provided in an embodiment of the present application.

[0030] Figure 4 It is a structural schematic diagram of a computer device provided by an optional embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0032] It should be understood that the "indication" mentioned in the embodiments of the present application can be a direct indication, an indirect indication, or an indication of an association relationship. For example, A indicates B, which can mean that A directly indicates B, for example, B can be obtained through A; it can also mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; it can also mean that there is an association relationship between A and B.

[0033] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0034] In an embodiment of the present application, "predefinition" can be achieved by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including a terminal device and a network device). The present application does not limit its specific implementation method.

[0035] First, the terms involved in this application are introduced.

[0036] Kubernetes: It is Google's open source container cluster management system, a distributed architecture solution based on container technology.

[0037] npd (node-problem-detector): Node problem detector is an important component in the Kubernetes ecosystem. It can run as a daemon to monitor and report the health status of nodes.

[0038] NVLink: A bus and communication protocol developed and launched by NVIDIA.

[0039] NVDEC5: A comprehensive set of APIs including high-performance tools, samples, and documentation for hardware-accelerated video encoding and decoding on Windows and Linux.

[0040] OFA (Optical Flow Accelerator): Optical flow accelerator, which is mainly responsible for providing necessary data for interpolation based on the motion vector information of geometric figures and special effects.

[0041] Apiserver: API server, a server program responsible for receiving and processing API requests in a distributed system.

[0042] Xid information: It is an error report printed by the Nvidia driver to the operating system kernel log or event log. Different Xids indicate different GPU problems. Therefore, you can capture Xid information to perceive GPU problems for subsequent processing.

[0043] Pod: It is the smallest deployment unit that can be created and managed in Kubernetes. It represents a group of containers running on the same node. These containers share the same network namespace and storage volume.

[0044] NVLink: A bus and communication protocol developed and launched by NVIDIA.

[0045] NVDEC5: A comprehensive set of APIs including high-performance tools, samples, and documentation for hardware-accelerated video encoding and decoding on Windows and Linux.

[0046] OFA (Optical Flow Accelerator): Optical flow accelerator, which is mainly responsible for providing necessary data for interpolation based on the motion vector information of geometric figures and special effects.

[0047] SPI: Serial Peripheral Interface.

[0048] PMU (Power Management Unit): A microcontroller that controls the power functions of a digital platform.

[0049] RPC (Remote Procedure Call): Remote procedure call.

[0050] NodeCondition: It is a mechanism to describe the status of a node. It is used to reflect the current health and availability of the node.

[0051] NoSchedule is a type of taint effect. When a node is assigned a taint and the NoSchedule effect is specified, it will prevent pods that have not declared tolerance for the taint from being scheduled on the node. This means that if there is no corresponding tolerance, the node will not be considered as a scheduling target for pods.

[0052] cuda kernel error: CUDA Kernel error is an error that may be encountered when using NVIDIA's CUDA (Compute Unified Device Architecture) for parallel computing programming.

[0053] Large models are huge in size and have extremely high requirements for network bandwidth, computing power and storage stability. They also require a stable and consistent operating environment. A large model can reach hundreds of GB or even TB levels, requiring a large-scale distributed training system and dedicated AI accelerators, which poses a huge challenge to the infrastructure. In order to achieve efficient training of large models, elastic distributed training has become an important means. It can flexibly schedule resources so that training tasks can be dynamically allocated to hundreds or thousands of nodes, significantly improving computing efficiency. Cloud native technology has built a large model infrastructure that supports elastic distributed training through containerization, microservices and dynamic orchestration. With the help of cloud native platforms, large models can flexibly access sufficient computing resources and achieve decoupling from the underlying infrastructure. Service discovery, load balancing, fault self-recovery and other capabilities at the platform level make large model deployment self-regulating and highly available. In addition, multi-tenant security isolation and resource management also allow large models to safely share cloud infrastructure. It can be said that cloud native technology provides an efficient and stable elastic distributed training environment for large models, which is an important support for the current large models from research to implementation. In order to enhance the fault detection and response capabilities of cloud native platforms at the node level, node-problem-detector (npd) came into being.

[0054] npd is an important component in the Kubernetes ecosystem. It comprehensively monitors the operation status of nodes by analyzing operating system logs and checking node process status. It can help identify node failures in a timely manner, such as memory anomalies, disk failures, etc., and report these failures to the Kubernetes Apiserver, so that Kubernetes can make scheduling decisions based on the node status to ensure high availability of services. The addition of npd makes Kubernetes more sensitive to node-level failures, and can respond quickly and self-repair node problems. It is one of the important tools to ensure stable and efficient training of large models. It provides a more robust node layer foundation for the cloud-native platform to support core application services such as upper-level large models.

[0055] Currently, implementing elastic distributed training for large models faces many challenges, such as discovery and coordination between training processes, member change management, and integration with existing code. The training of large models relies on GPUs, and Kubernetes's Apiserver cannot detect node anomalies caused by GPU failures in a timely manner by default, so instances may be scheduled to unavailable nodes during scheduling. At the same time, the existing elastic training frameworks cannot detect underlying GPU device errors, resulting in the user side being unable to stop losses in a timely manner.

[0056] Therefore, the embodiment of the present invention uses a node control method based on Kubernetes to detect node-level GPU failures through the component npd in large model distributed training. When a training node fails, npd can capture the fault log in time, modify the node status, and report it to the Apiserver. The Apiserver will taint the faulty node accordingly and adjust the training task allocation to only allocate tasks to healthy nodes to ensure training efficiency. The method flow of the node control method based on Kubernetes is as follows: Figure 1 As shown, the following steps are included.

[0057] S101. During the large-model distributed training process, the node fault log information is obtained through the node problem detection module of Kubernetes; the fault log information is used to characterize the fault type of the node; during the large-model distributed training process, corresponding training tasks are allocated to the callable nodes of the GPU.

[0058] Specifically, in step S101, large model distributed training relies on GPU, and the training tasks are assigned to the nodes corresponding to the GPU for training. Various failures may occur in the GPU during the training process. These failures may be caused by the hardware level, such as hardware damage, overheating, etc., or may be caused by the Nvidia software level or the upper application level. Various GPU problems may interrupt the training tasks on the current node. The Xid information is obtained through the npd (node ​​problem detector) component of Kubernetes. The Xid information is the error report printed by the Nvidia driver to the operating system kernel log or event log. Different Xid information is used to characterize different problems of the GPU. Therefore, the GPU problem can be perceived by capturing the Xid information for subsequent processing.

[0059] Optionally, in the above steps, the node problem detection module can customize the string pattern of each problem in the log in the configuration file, match the acquired fault log information according to the pattern, and a successful match indicates that the current problem has been located and needs to be processed. The node problem detection module can define these fault problems as temporary faults or permanent faults according to the settings, and then update the node status of the corresponding node according to the fault type.

[0060] S102. Based on the fault log information, update the node status of the node through the API server of Kubernetes; the node status includes whether the node is callable.

[0061] Specifically, in step S102, the node failure type includes temporary node failure and permanent node failure. Determine whether to shield the node according to the failure type of the GPU node. If the node is a temporary failure, set the node to be callable through the Kubernetes API server; if the node is a permanent failure, set the node to be uncallable through the Kubernetes API server. Ensure that training tasks will only be assigned to callable nodes, ensuring the training efficiency of large model distributed training.

[0062] Optionally, in step S102, although the nodes marked as permanent faults will not be called again in most cases, there is still a special case. After analyzing the fault log information, the non-callable nodes are added with taint labels, and training tasks will no longer be assigned to the taints unless the training tasks can tolerate the taint, to ensure that certain specific training tasks can be executed on nodes (i.e., tainted nodes) with limited resources, poor performance, or other potential problems.

[0063] Based on the above embodiments, the present application further provides a Kubernetes-based system, which includes a node problem detection module and an API server;

[0064] The node problem detection module is used to obtain the fault log information of the node during the large model distributed training process; the fault log information is used to characterize the fault type of the node; during the large model distributed training process, corresponding training tasks are allocated to the callable nodes of the GPU.

[0065] Specifically, the node problem detection module analyzes whether the training task is interrupted according to the fault log information; if the training task is interrupted, the training task is deleted through the Kubernetes API server.

[0066] The API server is used to update the node status of the node based on the fault log information; the node status includes whether the node is callable.

[0067] Specifically, the API server assigns the first training task among the training tasks to an uncallable node based on the node status of the node; a tolerance is set for the uncallable node; and the first training task is a training task among the training tasks that can meet the tolerance.

[0068] Optionally, the Kubernetes-based system also includes: a task scheduling module, which is used to assign the first training task in the training tasks to an uncallable node based on the node status of the node; the uncallable node is set with a tolerance; the first training task is a training task in the training tasks that can meet the tolerance.

[0069] In order to better illustrate the Kubernetes-based node control method and the Kubernetes-based system described in the above embodiment, a specific example will be used for illustration. The specific architecture of this example takes the Pytorch elastic training framework as an example. Figure 2 As shown, it includes: Fault-Tolerant-Controller (fault-tolerant controller), Node1 (node ​​1) and API Server (API server).

[0070] In large-model distributed training, the Xid information of the GPU node is obtained through the npd component of Kubernetes. Since this instance is applied to large-model distributed training, it mainly deals with device-level failures. Therefore, only the Xid information related to the hardware is perceived. Specifically, this article processes error reports with Xids of 31, 48, 56-58, 62-65, 68, 69, 70-77, 79-86, 88, 89, 95-105, 110-119, 122, and 125. As shown in Table 1, some Xids and corresponding error information are listed.

[0071] Table 1

[0072] Xid GPU failure information 74 NVLINK Failure 86 OFA Abnormal 96 NVDEC5 failure 122 SPIPMU RPC read error

[0073] Based on the specific content of the above Xid information, npd can define temporary and permanent problems in a customized way and report them in different ways. Specifically, the GPU failure with Xid information 31 is defined as a temporary problem, that is, a problem with little impact on the Pod on the current node, and is reported to Apiserver in the form of an event. The remaining Xid information mentioned above is defined as a permanent problem, the node status NodeCondition field is modified to GPUUnhealthy (GPU is unhealthy), and the taint NoSchedule is added to the node. The effect of this taint is that new Pods are not allowed to be scheduled to the current node unless the Pod can tolerate this taint.

[0074] Update the node status according to the node fault type obtained from the Xid information, update the GPU node status according to the fault log information, and then reschedule the training tasks according to the node status. This can ensure that the training tasks of large model distributed training will only be scheduled to healthy nodes, thereby significantly improving the training efficiency of large models. Figure 2 In the architecture shown, the specific process is as follows.

[0075] During the large-model distributed training process, npd obtains the task log information, analyzes the fault type from the task log information, and then modifies the nodecondition (node ​​status) according to the fault type, and adds taint labels to those permanently faulty nodes that cannot be called. Training tasks will no longer be assigned to these tainted nodes unless the training task can tolerate the taint to ensure that certain specific training tasks can be executed on nodes with limited resources, poor performance, or other potential problems (i.e., tainted nodes). Since npd matches the task log information according to the string pattern of each problem in the log, a successful match means that the current problem has been located and needs to be handled. If there is a string in the cuda kernelerror pattern in the task log information, it means that the training task in the Pod was indeed interrupted due to a GPU device failure. At this time, delete the current Pod.

[0076] The method provided in this example obtains GPU fault log information through the node problem detection module of Kubernetes, identifies multiple GPU faults and handles different faults in different ways, updates the status of GPU nodes according to the fault log information, and then reschedules the training tasks according to the node status, which can ensure that the training tasks of large model distributed training will only be scheduled to healthy nodes, thereby significantly improving the training efficiency of large models. Specifically, the method provided in this example is based on Nvidia's official GPU Xid fault log, which can identify multiple GPU faults from the system kernel log and handle different faults in different ways. It can improve Kubernetes's perception of node faults caused by GPU device failures and handle different fault types differently. By tainting permanent GPU nodes and rescheduling training tasks, it can ensure that model training tasks will only be scheduled to healthy nodes, thereby significantly improving the training efficiency of large models.

[0077] In summary, the node control method based on Kubernetes provided by the embodiment of the present invention is applied to the process of large model distributed training. Large model distributed training relies on GPU, and the training task is assigned to the node of GPU for distributed training. The method of the present invention detects the GPU node in the training process through the node problem detection module of Kubernetes, obtains the fault log information and sends it to the API server, the fault log information contains the fault type of the node, and the fault type includes temporary fault and permanent fault. The API server updates the state of the node by the node fault type indicated by the fault log information. If the node is a temporary fault, the state of the node is updated to be callable, and the training task can continue to be assigned to the node; if the node is a permanent fault, the state of the node is updated to be uncallable. After the node state becomes uncallable, the node will be stained, unless the training task can tolerate the stain, otherwise the training task will no longer be assigned to the node. The fault log information not only includes the fault type of the node, but also includes whether the training task is interrupted. If the training task is interrupted due to a GPU device failure, the current training task is deleted. The node problem detection module of Kubernetes is used to obtain GPU fault log information, identify various GPU faults, and handle different faults in different ways. The status of the GPU node is updated according to the fault log information, and the training tasks are rescheduled according to the node status. This ensures that the training tasks of large model distributed training will only be scheduled to healthy nodes, thereby significantly improving the training efficiency of large models.

[0078] In the embodiments of the present application, a node control device based on Kubernetes is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0079] The present application embodiment provides a node control device based on Kubernetes. Figure 3 : is a schematic diagram of the structure of a node control device based on Kubernetes provided in an embodiment of the present application, the device comprising:

[0080] The fault log acquisition module 301 is used to obtain the fault log information of the node through the node problem detection module of Kubernetes during the large model distributed training process; the fault log information is used to characterize the fault type of the node; during the large model distributed training process, corresponding training tasks are allocated to the callable nodes of the GPU.

[0081] The node status update module 302 is used to update the node status of the node through the API server of Kubernetes based on the fault log information; the node status includes whether the node is callable.

[0082] In an optional embodiment, the node status update module 302 is also used to: based on the node status of the node, assign the first training task in the training tasks to an uncallable node; the uncallable node is set with a tolerance; the first training task is a training task in the training tasks that can meet the tolerance.

[0083] In an optional embodiment, the Kubernetes-based node control device also includes: a task scheduling module 303, which is used to allocate the first training task in the training task to an uncallable node based on the node status of the node; the uncallable node is set with a tolerance; the first training task is a training task in the training task that can meet the tolerance.

[0084] In an optional implementation, the task scheduling module 303 is further used to: analyze whether the training task is interrupted according to the fault log information; if the training task is interrupted, delete the training task through the Kubernetes API server.

[0085] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0086] The Kubernetes-based node control device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0087] The embodiment of the present invention also provides a computer device having the above Figure 3 The node control device based on Kubernetes is shown.

[0088] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphic information in a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0089] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0090] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0091] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0092] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0093] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0094] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0095] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0096] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A node control method based on Kubernetes, characterized in that: The method comprises: During the large-model distributed training process, the fault log information of the node is obtained through the node problem detection module of Kubernetes; the fault log information is used to characterize the fault type of the node; during the large-model distributed training process, the corresponding training tasks are allocated to the callable nodes of the GPU; Based on the fault log information, the node status of the node is updated through the API server of Kubernetes; the node status includes whether the node is callable.

2. The node control method based on Kubernetes according to claim 1, characterized in that: The node failure types include temporary node failure and permanent node failure.

3. The node control method based on Kubernetes according to claim 2, characterized in that: The method further comprises: If the node is temporarily faulty, the node is set to be callable through the Kubernetes API server; if the node is permanently faulty, the node is set to be uncallable through the Kubernetes API server.

4. The node control method based on Kubernetes according to claim 1, characterized in that: The method further comprises: Based on the node status of the node, the first training task in the training tasks is assigned to the uncallable node; the uncallable node is set with a tolerance; the first training task is a training task in the training tasks that can meet the tolerance.

5. The node control method based on Kubernetes according to claim 1, characterized in that: The fault log is also used to indicate whether the training task is interrupted; if the training task is interrupted, the training task is deleted through the Kubernetes API server.

6. A Kubernetes-based system, characterized in that: The system includes a node problem detection module and an API server; The node problem detection module is used to obtain the fault log information of the node during the large model distributed training process; the fault log information is used to characterize the fault type of the node; During the large model distributed training process, corresponding training tasks are allocated to the callable nodes of the GPU; The API server is used to update the node status of the node based on the fault log information; the node status includes whether the node is callable.

7. The Kubernetes-based system according to claim 6, characterized in that: The system further comprises: A task scheduling module is used to allocate the first training task in the training tasks to an uncallable node based on the node status of the node; the uncallable node is set with a tolerance; the first training task is a training task in the training tasks that can meet the tolerance.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the Kubernetes-based node control method according to any one of claims 1 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the Kubernetes-based node control method according to any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the Kubernetes-based node control method according to any one of claims 1 to 5.

Images