Large model training task monitoring method and device, equipment, storage medium and product
By monitoring cluster resources and node status, obtaining and cleaning logs from large model training tasks, and performing visual monitoring, the problem of missing monitoring of the entire process of large model training tasks is solved, and full-process automated management and rapid fault location are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD
- Filing Date
- 2026-05-09
- Publication Date
- 2026-06-09
AI Technical Summary
The lack of existing technology for monitoring the entire training process of large models makes it difficult to effectively monitor and manage training tasks that can last for months.
The list-watch mechanism is used to monitor the status of custom resources and training task nodes in the cluster, obtain and clean the logs of the training tasks, obtain training task metrics, and finally perform visual monitoring.
It achieves automated monitoring of the entire process and lifecycle of large model training tasks, can accurately locate training anomalies, shorten the troubleshooting time, reduce operation and maintenance costs, and ensure the stable progress of training tasks.
Smart Images
Figure CN122173368A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium and product for monitoring large model training tasks. Background Technology
[0002] Large models refer to machine learning models with extremely large numbers of parameters and complex computational structures. These models are typically built from deep neural networks and have billions or even hundreds of billions of parameters. Training such models requires massive amounts of data and powerful computing resources. Monitoring the entire training process is a crucial aspect of large model training and cluster optimization. Currently, there is a lack of comprehensive monitoring solutions for large model training tasks that can take months. Summary of the Invention
[0003] This application provides a method, apparatus, device, storage medium, and product for monitoring large model training tasks, in order to address the lack of a monitoring solution for the entire process of large model training tasks in the prior art.
[0004] To achieve the above objectives, embodiments of this application provide a method for monitoring large model training tasks, including: The status of custom resources in the cluster is monitored using a list-watch mechanism; wherein, the custom resources are used to deploy and manage training tasks in the cluster. Use the list-watch mechanism to monitor the status of the nodes executing the training task; The name of the training task is obtained using the list-watch mechanism; Based on the name, retrieve the logs of the training task; The logs are cleaned to obtain training task metrics; The training task metrics are visualized.
[0005] As an improvement to the above solution, the method of using the resource monitoring list-watch mechanism to monitor the status of custom resources in the cluster includes: Based on the list-watch mechanism, the instance information of the custom resource is obtained; The status of the custom resource is monitored using a list-watch mechanism.
[0006] As an improvement to the above scheme, the step of using a list-watch mechanism to monitor the state of the node executing the training task includes: The agent deployed on the node executing the training task collects the status of the node executing the training task and reports the node fault information to the configuration resource configmap of the master node. Use the list-watch mechanism to monitor the configmap.
[0007] As an improvement to the above solution, the method further includes: When the status of the custom resource is detected as being interrupted during the training task, the training task is resumed from the breakpoint.
[0008] As an improvement to the above scheme, the step of cleaning the logs to obtain training task metrics includes: Obtain the log cleaning template corresponding to the training task; wherein the log cleaning template is obtained in advance by matching preset training task indicators using regular expressions; The logs are cleaned using the log cleaning template to obtain training task metrics.
[0009] As an improvement to the above scheme, the training task metrics include at least one of the following: Training steps, loss rate, single-card computing power, learning rate, and time per step.
[0010] To achieve the above objectives, embodiments of this application also provide a large model training task monitoring device, comprising: The first monitoring module is used to monitor the status of custom resources in the cluster using a resource monitoring list-watch mechanism; wherein, the custom resources are used to deploy and manage training tasks in the cluster; The second monitoring module is used to monitor the status of the nodes executing the training task using the list-watch mechanism; The first acquisition module is used to acquire the name of the training task using a list-watch mechanism; The second acquisition module is used to acquire the logs of the training task based on the name; The cleaning module is used to clean the logs to obtain training task metrics; The visualization module is used to visualize the training task metrics.
[0011] To achieve the above objectives, this application also provides a large model training task monitoring device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the large model training task monitoring method described above.
[0012] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the large model training task monitoring method described above.
[0013] To achieve the above objectives, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implements the large model training task monitoring method described above.
[0014] Compared with existing technologies, the present application provides a method, apparatus, device, storage medium, and product for monitoring large model training tasks. This method utilizes a resource monitoring list-watch mechanism to monitor the status of custom resources in a cluster. These custom resources are used to deploy and manage training tasks within the cluster. The list-watch mechanism is used to monitor the status of nodes executing the training tasks. The list-watch mechanism is used to obtain the name of the training task. Based on the name, the logs of the training task are obtained. The logs are cleaned to obtain training task metrics. The training task metrics are visualized. This enables multi-dimensional monitoring of the training task's running status, the status of training task nodes, and the training task metrics, thereby achieving automated monitoring of the entire process and lifecycle of large model training tasks. Attached Figure Description
[0015] Figure 1 This is a flowchart of a large model training task monitoring method provided in an embodiment of this application; Figure 2 This is a structural block diagram of a large model training task monitoring device provided in an embodiment of this application; Figure 3 This is a structural block diagram of a large model training task monitoring device provided in an embodiment of this application. Detailed Implementation
[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0017] In the description of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0018] In this application description, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0019] In this application description, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." The term "and / or" means at least one of the connected objects, such as A and / or B, indicating three cases: including only A, only B, and both A and B. Unless otherwise stated, the term "multiple" means two or more.
[0020] See Figure 1 , Figure 1 This is a flowchart of a large model training task monitoring method provided in an embodiment of this application. The large model training task monitoring method includes: S1. Use the resource monitoring list-watch mechanism to monitor the status of custom resources in the cluster; wherein, the custom resources are used to deploy and manage training tasks in the cluster; It is worth noting that a Custom Resource Definition (CRD) includes at least one of the following: PyTorchJob, AscendJob, or TFJob; this custom resource is used to deploy and manage training tasks in the cluster.
[0021] This application embodiment can utilize Java clients and Go clients, combined with the list-watch mechanism, to implement the status of custom resources in a Kubernetes cluster; listening to the status of custom resources is actually listening to the status of the controller corresponding to the custom resource.
[0022] Specifically, the use of the resource monitoring list-watch mechanism to monitor the status of custom resources in the cluster includes: Based on the list-watch mechanism, the instance information of the custom resource is obtained; The status of the custom resource is monitored using a list-watch mechanism.
[0023] For example, the client establishes a persistent connection with the Kubernetes cluster's API server. Initially, based on the list-watch mechanism (resource monitoring mechanism), the client retrieves instance information of custom resources from the API server. Subsequently, the API server records the status of the custom resources, such as when a new training task is created. When the client detects a change in the status of a custom resource, it records that change in the database.
[0024] For example, when a training task is forcibly interrupted, the status of a custom resource can be detected to change from MODIFY to DELETE, indicating that the training task has been deleted. After detecting the deletion of the training task, the information is reported to the alarm system.
[0025] S2. Use the list-watch mechanism to monitor the status of the nodes executing the training task; Specifically, the step of using the list-watch mechanism to monitor the state of the nodes executing the training task includes: The agent deployed on the node executing the training task collects the status of the node executing the training task and reports the node fault information to the configuration resource configmap of the master node. Use the list-watch mechanism to monitor the configmap.
[0026] It is worth noting that in this embodiment, the agent is deployed on the cluster. Specifically, the agent is deployed on the nodes executing the training tasks via daemonSets. The agent collects the status of each node and reports node fault information to the master node's configmap, which is then monitored using a list-watch mechanism. Furthermore, this node fault information can be reported to an alarm system for effective monitoring.
[0027] S3. Use the list-watch mechanism to obtain the name of the training task; It is worth noting that the list-watch mechanism obtains the name of the training task through information from the model training scheduling framework.
[0028] S4. Obtain the log of the training task according to the name; It's worth noting that training tasks are executed on pods (the smallest scheduling unit), so the logs for training tasks are actually pod logs. Since the training task name is associated with the pod name, the log path information for a specific pod can be obtained, thus automatically linking the training task to the training logs.
[0029] S5. Clean the logs to obtain training task metrics; Specifically, the step of cleaning the logs to obtain training task metrics includes: Obtain the log cleaning template corresponding to the training task; wherein the log cleaning template is obtained in advance by matching preset training task indicators using regular expressions; The logs are cleaned using the log cleaning template to obtain training task metrics.
[0030] It is worth noting that, in this embodiment of the application, a pre-configured log cleaning template corresponding to the training task can be configured by matching a preset training task using regular expressions. For example, if the log content is "10.11.129.1: [2024-03-08 23:13:10,249] [INFO][logging.py:77:log_dist] [Rank 0]step=77701, skipped=0, lr=[12,12], mom=[[0.9, 0.95], [0.9, 0.95]]", in order to obtain the training step (step) metric, the regular expression can be configured as {"field":"step","pattern":"step=(\d+)"}, indicating that the metric name is step, and the value of 77701 in the original file "step=77701" is extracted. Other metrics are processed in the same way. After configuration, the training task information and template information are saved to a MySQL database, thereby obtaining the log cleaning template corresponding to the training task through the MySQL database.
[0031] Optionally, the training task metrics include at least one of the following: Training steps, loss rate, single-card computing power, learning rate, and time per step.
[0032] For example, the component Loggie is deployed as a DaemonSet on a Kubernetes cluster to collect logs from training tasks. Furthermore, the collected logs can be published to the Kafka message queue. The Flink (stream and batch processing engine) program consumes the log messages from Kafka in real time, periodically loads log cleaning templates from the MySQL database into the cache, and parses key metrics of the training tasks, including: number of training steps for the large model, loss rate, total FLOPS (Total FLOPS), learning rate, and time per step.
[0033] S6. Visualize the training task metrics.
[0034] It is worth noting that this application provides a visual interface to visualize training task metrics, enabling observability of the training process and effective monitoring of training faults. For example, when an anomaly is detected in a certain training task metric, the anomaly information can be reported to the alarm system for alerting or in-depth analysis.
[0035] Furthermore, this application embodiment can also use Logstash (collection) to transform logs stored in Kafka into Elasticsearch (storage + search), and finally use Kibana (display) to provide visualization, retrieval and analysis functions for training logs.
[0036] The aforementioned training tasks specifically refer to large-scale model training tasks, which include, but are not limited to, training tasks for models such as image classification, object detection and recognition, and text classification. This application's embodiments do not impose specific limitations. For example, for the training task of an image classification model, a list-watch mechanism is used to monitor the training task's running status, node status, and metrics from multiple dimensions. This achieves automated monitoring of the entire process and lifecycle of the training task, not only accurately locating training anomalies and significantly shortening troubleshooting time, but also reducing the operational costs and computational waste of large-scale model training, ultimately ensuring the stable progress of the training task and classification accuracy.
[0037] In an optional embodiment, the method further includes: When the status of the custom resource is detected as being interrupted during the training task, the training task is resumed from the breakpoint.
[0038] It's worth noting that when a training task in progress is interrupted due to a hardware or software failure on a node in the cluster, the training task will first be resumed from its breakpoint. This means saving the current training task's data and metadata, and after the faulty node is resolved, a new node will be added to the cluster. This new node is a backup node in the cluster.
[0039] This application provides a method for monitoring large model training tasks. It utilizes a resource monitoring list-watch mechanism to listen to the status of custom resources in a cluster. These custom resources are used to deploy and manage training tasks within the cluster. The list-watch mechanism is used to monitor the status of nodes executing the training tasks. The list-watch mechanism is used to obtain the name of the training task. Based on the name, the logs of the training task are obtained. The logs are cleaned to obtain training task metrics. The training task metrics are visualized. This method enables multi-dimensional monitoring of the training task's running status, the status of training task nodes, and the training task metrics, thereby achieving automated monitoring of the entire process and lifecycle of large model training tasks.
[0040] See Figure 2 , Figure 2 This is a structural block diagram of a large model training task monitoring device 10 provided in an embodiment of this application. The large model training task monitoring device 10 includes: The first monitoring module 11 is used to monitor the status of custom resources in the cluster using a resource monitoring list-watch mechanism; wherein, the custom resources are used to deploy and manage training tasks in the cluster; The second monitoring module 12 is used to monitor the status of the nodes executing the training task using a list-watch mechanism; The first acquisition module 13 is used to acquire the name of the training task using a list-watch mechanism; The second acquisition module 14 is used to acquire the logs of the training task according to the name; Cleaning module 15 is used to clean the logs to obtain training task metrics; The visualization module 16 is used to visualize the training task metrics.
[0041] Optionally, the step of using the resource monitoring list-watch mechanism to monitor the status of custom resources in the cluster includes: Based on the list-watch mechanism, the instance information of the custom resource is obtained; The status of the custom resource is monitored using a list-watch mechanism.
[0042] Optionally, the step of using the list-watch mechanism to monitor the state of the node executing the training task includes: The agent deployed on the node executing the training task collects the status of the node executing the training task and reports the node fault information to the configuration resource configmap of the master node. Use the list-watch mechanism to monitor the configmap.
[0043] Optionally, the device further includes: The breakpoint resume training module is used to resume training on the training task when the status of the custom resource is detected as interrupted.
[0044] Optionally, the step of cleaning the logs to obtain training task metrics includes: Obtain the log cleaning template corresponding to the training task; wherein the log cleaning template is obtained in advance by matching preset training task indicators using regular expressions; The logs are cleaned using the log cleaning template to obtain training task metrics.
[0045] Optionally, the training task metrics include at least one of the following: Training steps, loss rate, single-card computing power, learning rate, and time per step.
[0046] It is worth noting that the working process of each module in the large model training task monitoring device 10 described in this application embodiment can refer to the working process of the large model training task monitoring method described in the above embodiment, and will not be repeated here.
[0047] This application provides a large model training task monitoring device 10 that uses a resource monitoring list-watch mechanism to monitor the status of custom resources in a cluster. These custom resources are used to deploy and manage training tasks within the cluster. The list-watch mechanism is used to monitor the status of nodes executing the training tasks. The list-watch mechanism is used to obtain the name of the training task. Based on the name, the logs of the training task are obtained. The logs are cleaned to obtain training task metrics. The training task metrics are visualized, enabling multi-dimensional monitoring of the training task's running status, the status of training task nodes, and the training task metrics. This achieves automated monitoring of the entire process and lifecycle of large model training tasks.
[0048] Furthermore, this application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the large model training task monitoring method as described in any of the above embodiments.
[0049] Furthermore, this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the large model training task monitoring method as described in any of the above embodiments.
[0050] See Figure 3 , Figure 3 This is a structural block diagram of a large model training task monitoring device 20 provided in an embodiment of this application. The large model training task monitoring device 20 includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described large model training task monitoring method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.
[0051] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the large model training task monitoring device 20.
[0052] The large model training task monitoring device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the large model training task monitoring device 20 and does not constitute a limitation on the device. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components. For example, the large model training task monitoring device 20 may also include input / output devices, network access devices, buses, etc.
[0053] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the large model training task monitoring device 20, connecting all parts of the large model training task monitoring device 20 via various interfaces and lines.
[0054] The processor 21 can be any one of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), NPU (Neural Network Processing Unit), DPU (Deep Learning Processing Unit), APU (Accelerated Processing Unit), and GPGPU (General-Purpose Computing on Graphics Processing Unit). The processor 21 is the control center of the large model training task monitoring device 20, connecting various parts of the electronic device via various interfaces and lines.
[0055] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the large model training task monitoring device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store relevant data, etc. In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.
[0056] The modules / units integrated in the large model training task monitoring device 20, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.
[0057] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0058] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.
Claims
1. A method for monitoring large model training tasks, characterized in that, include: The status of custom resources in the cluster is monitored using a list-watch mechanism; wherein, the custom resources are used to deploy and manage training tasks in the cluster. Use the list-watch mechanism to monitor the status of the nodes executing the training task; The name of the training task is obtained using the list-watch mechanism; Based on the name, retrieve the logs of the training task; The logs are cleaned to obtain training task metrics; The training task metrics are visualized.
2. The large model training task monitoring method as described in claim 1, characterized in that, The method of using the resource monitoring list-watch mechanism to monitor the status of custom resources in the cluster includes: Based on the list-watch mechanism, the instance information of the custom resource is obtained; The status of the custom resource is monitored using a list-watch mechanism.
3. The large model training task monitoring method as described in claim 1, characterized in that, The step of using the list-watch mechanism to monitor the state of the nodes executing the training task includes: The agent deployed on the node executing the training task collects the status of the node executing the training task and reports the node fault information to the configuration resource configmap of the master node. The configmap is monitored using a list-watch mechanism.
4. The large model training task monitoring method as described in claim 1, characterized in that, The method further includes: When the status of the custom resource is detected as being interrupted during the training task, the training task is resumed from the breakpoint.
5. The large model training task monitoring method as described in claim 1, characterized in that, The process of cleaning the logs to obtain training task metrics includes: Obtain the log cleaning template corresponding to the training task; wherein the log cleaning template is obtained in advance by matching preset training task indicators using regular expressions; The logs are cleaned using the log cleaning template to obtain training task metrics.
6. The large model training task monitoring method as described in claim 1, characterized in that, The training task metrics include at least one of the following: Training steps, loss rate, single-card computing power, learning rate, and time per step.
7. A monitoring device for large model training tasks, characterized in that, include: The first monitoring module is used to monitor the status of custom resources in the cluster using a resource monitoring list-watch mechanism; wherein, the custom resources are used to deploy and manage training tasks in the cluster; The second monitoring module is used to monitor the status of the nodes executing the training task using the list-watch mechanism; The first acquisition module is used to acquire the name of the training task using a list-watch mechanism; The second acquisition module is used to acquire the logs of the training task based on the name; The cleaning module is used to clean the logs to obtain training task metrics; The visualization module is used to visualize the training task metrics.
8. A monitoring device for large model training tasks, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the large model training task monitoring method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the large model training task monitoring method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the large model training task monitoring method as described in any one of claims 1 to 6.