Deep learning training task state monitoring method and device, equipment and medium

Abstract

The embodiment of the invention provides a deep learning training task state monitoring method and device, equipment and a medium, and relates to the technical field of computers. The method comprises the steps that after a machine learning training task is submitted to a container arrangement engine, multi-level operation state data of the container arrangement engine are acquired; performing state verification based on the multi-level operation state data according to a preset state verification sequence to obtain a plurality of corresponding state verification results; and integrating the plurality of state verification results to determine a business state judgment result corresponding to the machine learning training task. According to the embodiment of the invention, the multi-level state monitoring data of the container arrangement engine is obtained, the states of the multiple levels are verified, and finally the verification results are integrated and the task running state is comprehensively judged, so that the accuracy of model training task state monitoring is effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, device, and medium for state monitoring of deep learning training tasks. Background Technology

[0002] In the field of machine learning and model training, it is necessary to monitor the status of the allocation and scheduling of computing resources for training tasks, so as to provide business systems with the function of querying and managing the status of training tasks.

[0003] Currently, the business system primarily determines the running status of training tasks by obtaining key status fields output by the container orchestration engine. For example, the business system monitors the "active" status field of the Job object in real time through a monitoring mechanism deployed on the container orchestration engine; when the value of this status field is greater than 0, the task is considered to be in a running state.

[0004] However, the status field value can only roughly determine the running status of a task. In many scenarios, the actual running status of a task will not match the rough judgment result. Therefore, the existing task running status monitoring solution has the problem of low accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, device, and medium for monitoring the state of deep learning training tasks, so as to improve the accuracy of task operation state monitoring.

[0006] In a first aspect, embodiments of this application provide a state monitoring method for a deep learning training task, including: After submitting the machine learning training task to the container orchestration engine, obtain the multi-level runtime status data of the container orchestration engine; According to the preset state verification order, state verification is performed based on the multi-level operating state data to obtain multiple corresponding state verification results. The multiple state verification results are integrated to determine the business state judgment result corresponding to the machine learning training task.

[0007] In this embodiment, by acquiring multi-level state monitoring data of the container orchestration engine and performing state verification on multiple levels respectively, the verification results are integrated and the task running status is comprehensively determined, thereby effectively improving the accuracy of state monitoring of the model training task.

[0008] In some embodiments, obtaining the multi-level runtime status data of the container orchestration engine includes: Obtain the scheduling determination result of the scheduling component in the container orchestration engine; If the scheduling determination result indicates that the scheduling is successful, the multi-level status monitoring engine is activated and the multi-level running status data of the container orchestration engine is obtained.

[0009] In this embodiment, the accuracy of training task status monitoring is further improved by determining whether multi-level fine-grained verification is triggered based on the scheduling status of the scheduling layer.

[0010] In some embodiments, the step of performing state verification based on the multi-level operating state data according to a preset state verification order to obtain multiple corresponding state verification results includes: Obtain the Pod layer resource allocation information corresponding to the container orchestration engine, and perform status verification based on the Pod layer resource allocation information to obtain the corresponding first status verification result; The first status verification result is used to characterize whether the Pod resource allocation result meets the preset allocation success conditions.

[0011] In this embodiment of the application, the accuracy of training task status monitoring is further improved by verifying the resource allocation status of the Pob layer to determine whether the Pod resource allocation meets the preset conditions.

[0012] In some embodiments, the state monitoring method for the deep learning training task further includes: If the preset allocation success condition is met based on the first status verification result, the running status information of the Pod layer corresponding to the container orchestration engine is obtained, and the status verification is performed based on the running status information of the Pod layer to obtain the corresponding second status verification result. The second status verification result is used to characterize whether there are any abnormalities in the running status of each Pod during its running phase.

[0013] In this embodiment of the application, the running status of the Pod layer is verified to confirm whether there are any abnormalities in the running status of each Pod running stage, thereby further improving the accuracy of training task status monitoring.

[0014] In some embodiments, after obtaining the Pod layer running status information corresponding to the container orchestration engine, the method further includes: If it is determined that the currently running task requires GPU resources, GPU layer status monitoring data is obtained, and status verification is performed based on the GPU layer status monitoring data to obtain the corresponding third status verification result. The third state verification result is used to characterize whether the actual resource allocation record of the container orchestration engine matches the resource allocation information reported by the hardware allocation management component.

[0015] In this embodiment, the GPU layer status is verified to determine whether the actual resource allocation of the container orchestration engine matches the reported information, thereby further improving the accuracy of training task status monitoring.

[0016] In some embodiments, integrating the multiple state verification results to determine the business state judgment result corresponding to the machine learning training task includes: If all the status verification results indicate successful verification, the business status determination result is determined to be normal operation; If any one of the multiple status verification results indicates a verification failure, the business status determination result is determined to be abnormal.

[0017] In this embodiment, by integrating the verification results of each level, the task is determined to be running normally only when all the verification results of each level pass successfully, thereby further improving the reliability of training task status monitoring.

[0018] In some embodiments, the state monitoring method for the deep learning training task further includes: If the business status determination result is determined to be abnormal, the operation abnormality location information is determined based on each of the status verification results.

[0019] In the embodiments of this application, when the verification results of each level are integrated and a comprehensive judgment is made that an anomaly is made, the anomaly information is located by the verification situation of each level, thereby further improving the accuracy and efficiency of fault location of anomaly status.

[0020] In some embodiments, the state monitoring method for the deep learning training task further includes: If the scheduling determination result indicates a scheduling failure, the task scheduling circuit breaker mechanism is triggered, and the scheduling layer anomaly location information is output.

[0021] In this embodiment of the application, when a scheduling failure occurs at the scheduling layer, the circuit breaker mechanism is directly triggered and the abnormal information of the scheduling layer is fed back, which further improves the efficiency of fault location of abnormal status.

[0022] Secondly, embodiments of this application provide a state monitoring device for a deep learning training task, comprising: The status monitoring module is used to obtain multi-level runtime status data of the container orchestration engine after submitting the machine learning training task to the container orchestration engine; The status verification module is used to perform status verification based on the multi-level operating status data according to a preset status verification order, and obtain multiple corresponding status verification results. The aggregation and determination module is used to integrate the multiple state verification results to determine the business state determination result corresponding to the machine learning training task.

[0023] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method described in any embodiment of the first aspect.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0025] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a state monitoring method for a deep learning training task provided in an embodiment of this application; Figure 2 A schematic diagram of the state monitoring process for deep learning training tasks in existing technologies; Figure 3 This is a schematic diagram of the deep learning training task state monitoring process provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of a state monitoring device for a deep learning training task provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0029] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] It should be noted that deep learning training typically requires a large amount of dedicated hardware resources such as GPUs. Kubernetes (K8s, a container orchestration engine) can manage heterogeneous computing resources in a unified manner through device plugins and automatically schedule tasks to appropriate nodes.

[0031] like Figure 2 As shown, the existing training task operation status monitoring process is as follows: First, users submit training tasks (single-machine / distributed) according to their needs, triggering a request to create a Job or CRD (such as TFJob). In the Kubernetes ecosystem, a Job is a workload object used to run one-time, short-term tasks, and is a standard abstraction for encapsulating and managing deep learning training tasks. CRD (Custom Resource Definition) is the core mechanism in Kubernetes used to extend the API.

[0032] Then, the Kube-API-Server receives requests for Jobs or CRDs. The Job Controller listens for events from the API Server and creates corresponding Pods based on the number of replicas. After the Pods are scheduled to GPU nodes, the GPU Device Plugin on the node is responsible for allocating device resources.

[0033] Finally, the Informer (list-watch mechanism) deployed in the business system starts listening to the status fields of the Job object. By checking the status.active field of the Job, when its value is >0, it is considered that the task is in a running state. At this time, it will further query the GPU resource declaration of the Pod (such as the nvidia.com / gpu field) to determine the GPU allocation status.

[0034] However, the aforementioned traditional condition monitoring scheme has at least the following drawbacks: 1. Prone to misjudgment of status: status.active > 0 only indicates that the Pod associated with the Job has been created, but the actual status of the Pod may be ImagePullBackOff (image pull failed), ContainerCreating (container starting), CrashLoopBackOff (crash loop), GPU not actually allocated or failed to mount, etc., and it is not actually running.

[0035] 2. GPU Invisibility: Only resource declarations are checked; the actual runtime GPU allocation and whether the allocation was successful are not verified.

[0036] To address the problems existing in the prior art, this application provides a state monitoring method for deep learning training tasks. By replacing single state judgment with a multi-level state verification mechanism, the probability of state misjudgment can be effectively reduced, thereby effectively improving the accuracy and reliability of state monitoring.

[0037] like Figure 1 As shown in the figure, this application provides a state monitoring method for a deep learning training task, which may include the following steps: S1. After submitting the machine learning training task to the container orchestration engine, obtain the multi-level runtime status data of the container orchestration engine.

[0038] like Figure 3 As shown, specifically, users can submit machine learning training tasks to a container orchestration engine (such as Kubernetes, or K8s for short) through their business systems. The system will then convert the user's training intent (code, data, resources) into resource definitions (YAML) that Kubernetes can recognize and execute, and submit them to the K8s API Server.

[0039] For example, the system can submit deep learning training task descriptions as needed, including single-machine training tasks or training tasks such as TF (TensorFlow distributed training task), PT (PyTorch distributed training task), and MPI. After receiving the task, the system automatically creates corresponding job resources in the Kubernetes cluster, such as standard Job resources or custom resource definitions (CRDs) for specific frameworks, such as TFJob, PyTorchJob, and MPIJob. At this stage, only the declaration of computing resources (such as the number of GPUs) is completed; actual scheduling and resource binding have not yet been performed.

[0040] Then, business systems can obtain multi-level runtime status data of the container orchestration engine in real time through relevant interfaces. These multiple levels can include Pod resource allocation verification, Pod runtime status monitoring, and GPU layer consistency verification.

[0041] S2. Perform status verification based on multi-level operating status data according to the preset status verification order to obtain multiple corresponding status verification results.

[0042] For example, multi-level status verification can be performed in the order of Pod resource allocation, runtime status monitoring, and then GPU layer to obtain the status verification results corresponding to each level.

[0043] S3. Integrate multiple status verification results to determine the business status judgment result corresponding to the machine learning training task.

[0044] Finally, the system integrates the status verification results from multiple levels and comprehensively determines the business status of the machine learning training task. For example, when all levels of verification meet preset conditions, the task's business status is marked as RUNNING (normal operation). If any level of verification fails, it is judged as abnormal, and the corresponding circuit breaker mechanism is triggered. The circuit breaker mechanism is a system protection mode used to prevent cascading failures and improve system resilience. When a service fails, it temporarily rejects subsequent requests to avoid exhausting system resources and causing the failure to spread.

[0045] Based on this, by acquiring multi-level state monitoring data from the container orchestration engine and performing state verification on multiple levels respectively, and finally integrating these verification results to comprehensively determine the task running status, the accuracy of state monitoring for model training tasks can be effectively improved.

[0046] In some embodiments, obtaining multi-level runtime status data of the container orchestration engine includes: Obtain the scheduling decision result from the scheduling component in the container orchestration engine; If the scheduling decision result indicates that the scheduling is successful, the multi-level status monitoring engine is activated and the multi-level running status data of the container orchestration engine is obtained.

[0047] like Figure 3 As shown, after the system creates the corresponding job resources, it first performs a status check at the scheduling layer. At the scheduling layer, the scheduling component (kube-scheduler) in Kubernetes makes scheduling decisions based on resource availability and outputs two results: 1. Scheduled (scheduled successfully): Records the target node information and activates subsequent multi-level status monitoring; 2. Unschedulable (scheduled failed): Circuit breaker is triggered within seconds, directly terminating the process.

[0048] It's important to note that kube-scheduler is the core scheduling component of Kubernetes, responsible for making scheduling decisions based on cluster status and Pod resource requirements. Each job has its own resource requirements, such as CPU, GPU, and memory. kube-scheduler determines which server each job will run on based on the availability of these resources. If the resource requirements are met, the job will be successfully scheduled, and the target node will be recorded—the server location where the job will run.

[0049] Understandably, this layer serves as the first verification checkpoint for the proposed solution. The system monitors the event stream of the Kubernetes API Server to capture scheduling events issued by kube-scheduler for the target Pod in real time. When a Scheduled event is detected, it indicates that the scheduler has successfully made a decision. The system records the target node information of the Pod and activates the subsequent resource allocation verification process. When a FailedScheduling event is detected, it indicates that the scheduling has failed.

[0050] It should be noted that the overall training task has a possibility of running successfully only when the scheduling layer determines that the scheduling is successful. Therefore, subsequent multi-level state monitoring is activated only when the scheduling is successful; otherwise, the circuit breaker mechanism is triggered directly.

[0051] Based on this, the accuracy of training task status monitoring is further improved by determining whether multi-level fine-grained verification is triggered according to the scheduling status of the scheduling layer.

[0052] It should be noted that in the event of a scheduling failure, the system will immediately trigger a second-level circuit breaker mechanism, terminating the entire process and returning the failure result to the user system. This mechanism achieves millisecond-level fault response by capturing decision events rather than waiting for state synchronization, thus avoiding resource idleness.

[0053] In traditional solutions, Informer's event processing chain is relatively long, with the state change propagation path being: Kubelet (node ​​agent) -> API Server -> Informer (client). Each step requires serialization, network transmission, queuing, and processing, resulting in a state synchronization delay of approximately 3-15 seconds and high latency in state monitoring.

[0054] Specifically, the condition monitoring process of the traditional solution is as follows: 1. kube-scheduler decides to bind the Pod to server A (Node A); 2. kube-scheduler sends a Binding request to the API Server, and the API Server updates the Pod.spec.nodeName field to "NodeA".

[0055] 3. The API Server writes the entire updated Pod object to Etcd. Only then is the scheduling result truly "written to disk".

[0056] 4. Etcd update events are notified to the API Server.

[0057] 5. The business system's Informer (List-Watch) retrieves the latest Pod list from the API Server and identifies changes.

[0058] The status monitoring process of this application is as follows: 1. kube-scheduler decides to bind the Pod to Node A (or scheduling fails); 2. The kube-scheduler immediately and simultaneously sends a request to the API Server to create an Event resource with the reason field set to Scheduled. This operation is typically performed in parallel with or immediately following the binding request in step 1.

[0059] 3. The business system client listens to the event listener of the Event resource, capturing the newly created Event almost in real time from the API Server's Watch channel. Based on this, the creation of the Event resource does not need to wait for Pod information to be persisted to Etcd, effectively reducing latency.

[0060] As can be seen, in traditional solutions, a Pod that fails to schedule due to insufficient resources will remain in a Pending state until it times out and is deleted. During this period, the business system will think that the task is "being scheduled", resulting in the resource request being suspended. However, the embodiment of this application achieves immediate circuit breaking by capturing the FailedScheduling event, which can clearly inform the user of the failure result within seconds. The system no longer needs to waste any monitoring resources for this task that is destined to fail, achieving the effect of quickly confirming failure and avoiding the illusion of idle resources.

[0061] In some embodiments, state verification is performed based on multi-level operating state data according to a preset state verification order to obtain multiple corresponding state verification results, including: Obtain the Pod layer resource allocation information corresponding to the container orchestration engine, and perform status verification based on the Pod layer resource allocation information to obtain the corresponding first status verification result; The first status verification result is used to characterize whether the Pod resource allocation result meets the preset allocation success conditions.

[0062] It should be noted that after the scheduling is successful and the target node is recorded, the status monitoring engine begins to verify the resource allocation of the Pod layer. This layer is responsible for verifying that the GPU resources have been successfully mapped from logical to physical on the target node.

[0063] Through the asynchronous listening mechanism, the system can start an asynchronous listener for the Pod to continuously monitor its status updates.

[0064] Its core objective is to wait for the status.containerStatuses[*].allocatedResources field to be correctly populated (e.g., showing nvidia.com / gpu: "1"), which is the authoritative record of successful resource allocation at the Kubernetes level.

[0065] While confirming the success of the resource allocation logic, the system extracts and records a list of globally unique identifiers (UUIDs) of the specific GPU devices allocated by DevicePlugin from the Pod's annotations or status, providing benchmark data for the final consistency verification.

[0066] Optionally, a timeout period (e.g., 30 seconds) can be set during the monitoring process at this layer. If the timeout occurs or a clear allocation failure event is detected, the circuit breaker mechanism will be triggered immediately.

[0067] Based on this, the accuracy of training task status monitoring is further improved by verifying the resource allocation status of the Pob layer to determine whether the Pod resource allocation meets the preset conditions.

[0068] In some embodiments, the state monitoring method for deep learning training tasks further includes: If the preset allocation success conditions are met based on the first state verification result, the running state information of the Pod layer corresponding to the container orchestration engine is obtained, and the state verification is performed based on the running state information of the Pod layer to obtain the corresponding second state verification result. The second state verification result is used to characterize whether there are any abnormalities in the running state of each Pod during its running phase.

[0069] Specifically, when the first status verification result indicates that the Pod's running status meets the preset successful allocation conditions, further status monitoring is performed based on the Pod layer's running status information to obtain the corresponding second status verification result. This layer starts after resource allocation confirmation and aims to track the startup and running status of the Pod's contents to ensure the successful operation of task instances.

[0070] It should be noted that at the Pod layer, the Pod Informer can be used to listen to and parse the Pod's status fields, and track the container lifecycle in real time: every sub-state change from Pending (waiting for scheduling) to Running (running) [such as Pulling (pulling the image) and ContainerCreating (creating the container)], thereby accurately identifying the blocking points in the startup process.

[0071] For example, when the target Pod (Container) changes its state to running, a runtime status liveness detection process is immediately triggered. The system executes low-overhead commands within the container via a proxy deployed on the target node and the container runtime interface to check the availability of processes and GPU resources within the container. For example, it verifies whether the main process of the training task (such as a Python process) is alive and whether the GPU environment is available. At the same time, it can obtain a list of UUIDs of the actual accessible GPU devices within the container.

[0072] This step prevents inconsistencies between the actual task status and the actual situation, such as "the container is running but the application has not started" or "GPU driver is abnormal".

[0073] Based on this, by verifying the running status of the Pod layer, it is possible to confirm whether there are any abnormalities in the running status of each Pod during its running phase, thereby further improving the accuracy of training task status monitoring.

[0074] In some embodiments, after obtaining the Pod layer runtime status information corresponding to the container orchestration engine, the method further includes: If it is determined that the currently running task requires GPU resources, GPU layer status monitoring data is obtained, and status verification is performed based on the GPU layer status monitoring data to obtain the corresponding third status verification result. The third state verification result is used to characterize whether the actual resource allocation record of the container orchestration engine matches the resource allocation information reported by the hardware allocation management component.

[0075] Specifically, after performing Pod-level state monitoring, it determines whether to enter the GPU-level verification based on whether the current task requires GPU resources. If the Pod requires GPU resources, it enters the GPU-level for state verification; if the Pod does not require GPU resources, it only enters the container lifecycle tracking and verification.

[0076] When performing state verification at the GPU layer, GPU allocation event information can be pushed through the Device Plugin (a core extension mechanism in Kubernetes used to manage and expose special hardware devices on nodes, such as GPUs, FPGAs, and InfiniBands, for use by containers). This information is then used to verify the mapping relationship between physical devices and logical resources. Based on this, it can be confirmed that the Kubernetes resource scheduling layer has successfully allocated the GPU device to the Pod and that it corresponds to the actual device on the node.

[0077] Specifically, the list of GPU UUIDs allocated at the cluster level, recorded by the L1 layer (resource allocation verification), is precisely compared with the list of GPU UUIDs actually identified from the container's runtime environment by the L2 layer (Pod runtime status monitoring). For example, if the two UUID lists are completely identical, it indicates that the GPU resources have been accurately allocated and bound to the training task, and the system ultimately determines the task status as successful. If they are inconsistent, it indicates an anomaly such as "device mapping deviation," and the system determines the allocation has failed.

[0078] For example, the GPU layer state verification process is as follows: 1. When allocating GPUs, the Device Plugin reports the allocated device information to Kubernetes; 2. Obtain the events reported in these reports and extract resource allocation information from them, including: the GPU device ID (physical device) allocated to the Pod, the unique identifier of the Pod (namespace, Pod name), the name of the allocated logical resource (such as nvidia.com / gpu), and the quantity, etc.

[0079] 3. Check whether the actual resource allocation records of the Pod in Kubernetes (recorded by the L1 layer) are consistent with the information reported by DevicePlugin (recorded by the L2 layer).

[0080] Based on this, by verifying the GPU layer status, it is possible to determine whether the actual resource allocation of the container orchestration engine matches the reported information, thereby further improving the accuracy of training task status monitoring.

[0081] In some embodiments, multiple state verification results are integrated to determine the business state judgment result corresponding to the machine learning training task, including: If multiple status verification results indicate successful verification, the business status is determined to be normal operation. If any one of the multiple status verification results indicates a verification failure, the business status determination result is determined to be abnormal.

[0082] For example, the final business status can be divided into two types: normal operation and normal operation. Based on this, the task is determined to be normal operation if and only if the status verification results at each level are all characterized as successful verification. Otherwise, if any status verification result is characterized as verification failure, an anomaly is determined to have occurred.

[0083] For example, for the L1 layer (resource allocation verification), if the first state verification result indicates that the Pod resource allocation result meets the preset allocation success conditions, then the L1 layer is determined to have passed the verification successfully; otherwise, the L1 layer verification is determined to have failed. For the L2 layer (running status monitoring), if the second state verification result indicates that the running status of each Pod in the running stage is not abnormal, then the L2 layer is determined to have passed the verification successfully; otherwise, the L2 layer verification is determined to have failed. For the L3 layer (GPU layer), if the third state verification result indicates that the actual resource allocation record of the container orchestration engine matches the resource allocation information reported by the hardware allocation management component, then the GPU layer is determined to have passed the verification successfully; otherwise, the GPU layer verification is determined to have failed.

[0084] Based on this, by integrating the verification results at all levels, the task is determined to be running normally only when all verification results at all levels pass successfully, thereby further improving the reliability of training task status monitoring.

[0085] In some embodiments, the state monitoring method for deep learning training tasks further includes: If the business status determination result is determined to be abnormal, the abnormality location information is determined based on the verification results of each status.

[0086] It should be noted that when the final business status determination result is abnormal, the fault location information can be located based on the verification details represented by each status verification result (determining the location information of the operational anomaly).

[0087] For example, the success or failure of each layer of verification directly corresponds to a specific fault domain. A failure at layer L0 is attributed to "insufficient cluster scheduling resources or policy conflicts"; a failure or timeout at layer L1 is attributed to "abnormal node GPU resource allocation components (such as Device Plugin) or device preparation failure"; and a failure at layer L2 is attributed to "abnormal container image, application code, or node runtime environment". This granular fault localization narrows the scope of troubleshooting for operations personnel from "the entire system" to "specific components," greatly improving operational efficiency and enabling rapid fault isolation and recovery.

[0088] Based on this, when the verification results of each level are integrated and a comprehensive judgment is made that an anomaly is found, the anomaly information is located by checking the verification status of each level, thereby further improving the accuracy and efficiency of fault location for anomalies.

[0089] In some embodiments, the state monitoring method for deep learning training tasks further includes: If the scheduling determination result indicates a scheduling failure, the task scheduling circuit breaker mechanism is triggered, and the scheduling layer anomaly location information is output.

[0090] It should be noted that since the final business state can only be running successfully if the scheduling is successful, if the scheduling determination result is characterized as scheduling failure at the scheduling layer, the abnormal location information of the current business abnormality can be directly determined as a scheduling layer abnormality.

[0091] Based on this, when a scheduling failure occurs at the scheduling layer, the circuit breaker mechanism is directly triggered and the abnormal information of the scheduling layer is fed back, which further improves the efficiency of fault location for abnormal states.

[0092] Compared with the prior art, the embodiments of this application have the following beneficial effects: 1. The accuracy of state judgment is effectively improved: By replacing the single state judgment with a four-layer independent verification mechanism, the misjudgment of "false state" and "false assignment" is effectively avoided, and the misjudgment rate is greatly reduced.

[0093] Specifically, existing technologies rely solely on querying macroscopic status fields (such as `status.active`) of Jobs or Pods, failing to penetrate and perceive the underlying actual situation. This application's solution achieves comprehensive status perception by constructing a full-stack, progressive verification chain from cluster scheduling (L0 layer) to hardware devices (L3 layer). Specifically, the L0 layer (scheduling layer) confirms successful "scheduling decisions" through event listening, avoiding misjudging "scheduling failure" as "scheduling in progress" due to API synchronization delays; the L1 layer (resource allocation confirmation) verifies the `allocatedResources` field and node device files to ensure that Kubernetes-level resource allocation is actually implemented, resolving the false allocation problem of "resources declared but not mounted"; the L2 layer (Pod runtime status monitoring) directly verifies the availability of business processes and the GPU environment through container liveness detection, resolving the false runtime problem of "containers running but application malfunctioning"; and the final L3 layer (GPU layer) provides irrefutable evidence of the accuracy of resource binding through UUID consistency verification. This cross-verification of multiple evidence chains improves the reliability of status judgment by orders of magnitude.

[0094] 2. Through full-stack monitoring capabilities from scheduling to hardware, accurate fault location and rapid isolation can be achieved.

[0095] Traditional solutions require operations personnel to sequentially troubleshoot multiple components, including the scheduler, Device Plugin, container runtime, and node drivers, when a task fails, resulting in low efficiency. This application's layered verification architecture naturally places monitoring points at each critical stage of the task lifecycle. Specifically, the success or failure of each layer of verification directly corresponds to a clear fault domain: L0 layer failure is attributed to "insufficient cluster scheduling resources or policy conflicts"; L1 layer failure or timeout is attributed to "abnormal node GPU resource allocation components (such as Device Plugin) or device preparation failure"; and L2 layer failure is attributed to "abnormal container image, application code, or node runtime environment". This refined fault localization narrows the scope of operations personnel's investigation from the "entire system" to "specific components," greatly improving operational efficiency and enabling rapid fault isolation and recovery.

[0096] 3. Achieve millisecond-level fault response, significantly improving system efficiency and user experience.

[0097] Because existing technologies rely on Informer polling mechanisms, state awareness latency can be as high as seconds. This application's embodiment introduces an event-driven mechanism in the most critical task startup phase. Specifically, in the scheduling stage (L0 layer), where task submission is most likely to fail, the system listens for the `FailedScheduling` event and can detect failure and immediately trigger a circuit breaker within milliseconds of the scheduler's decision. This prevents tasks from remaining in a meaningless "Pending" state for extended periods, saving user waiting time and releasing logical resources (such as quotas) occupied by the task request, allowing them to be used by other tasks, thereby improving the overall cluster resource utilization efficiency.

[0098] 4. The non-invasive design ensures the versatility, stability, and low cost of the solution.

[0099] Some existing alternatives (such as modifying CRDs or implanting probes within containers) can break Kubernetes compatibility or increase application overhead. The proposed solution, however, cleverly utilizes Kubernetes' existing extension mechanisms and standard interfaces. Specifically, the entire solution is based on listening to standard events, querying standard APIs, and executing standard container runtime commands through DaemonSet, without requiring modifications to the Kubernetes core code or training image. This non-intrusive design ensures seamless deployment on any standard Kubernetes cluster, maintaining excellent versatility. Furthermore, the verification action is triggered only during critical phases of task startup, rather than continuous high-frequency polling, resulting in minimal performance overhead on nodes and containers and guaranteeing production environment stability. In summary, this application addresses the issue of inaccurate status through multi-layered collaborative verification, the difficulty in fault localization through layered architecture, the slow response through event-driven approaches, and the difficulty in implementation through non-intrusive design. These effects are interconnected and work together to ultimately provide users with a highly reliable, efficient, and easy-to-maintain deep learning training task management solution.

[0100] Please refer to Figure 4 , Figure 4 The diagram illustrates a block diagram of a state monitoring apparatus for a deep learning training task provided in some embodiments of this application. It should be understood that this state monitoring apparatus for a deep learning training task is similar to the one described above. Figure 1 Corresponding to the method embodiments, it is able to execute each step involved in the above method embodiments. The specific functions of the state monitoring device for the deep learning training task can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here.

[0101] Figure 4 The state monitoring device for a deep learning training task includes at least one software functional module that can be stored in a memory or embedded in the state monitoring device for the deep learning training task in the form of software or firmware. The state monitoring device for the deep learning training task includes: The status monitoring module 410 is used to obtain multi-level runtime status data of the container orchestration engine after submitting the machine learning training task to the container orchestration engine. The status verification module 420 is used to perform status verification based on multi-level operating status data according to a preset status verification order, and obtain multiple corresponding status verification results. The aggregation and determination module 430 is used to integrate multiple status verification results to determine the business status determination result corresponding to the machine learning training task.

[0102] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention. The state monitoring device for a deep learning training task provided by the embodiments of the present invention can implement the state monitoring method for a deep learning training task provided by any one of the method embodiments of the present invention.

[0103] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0104] like Figure 5As shown, some embodiments of this application provide an electronic device 500, which includes: a memory 510, a processor 520, and a computer program stored in the memory 510 and executable on the processor 520. When the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement any of the methods included in the above-described deep learning training task state monitoring method.

[0105] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0106] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of this disclosure embodiment can be used to execute the instructions in the memory 510 to implement the methods shown above. The memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0107] Some embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, describes the method described in the method embodiments.

[0108] Some embodiments of this application also provide a computer program product that, when run on a computer, causes the computer to perform the method described in the method embodiments.

[0109] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0110] It should be understood, in the several embodiments provided in this application, that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0111] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0112] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0114] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A state monitoring method for a deep learning training task, characterized in that, include: After submitting the machine learning training task to the container orchestration engine, obtain the multi-level runtime status data of the container orchestration engine; According to the preset state verification order, state verification is performed based on the multi-level operating state data to obtain multiple corresponding state verification results. The multiple state verification results are integrated to determine the business state judgment result corresponding to the machine learning training task.

2. The state monitoring method for deep learning training tasks according to claim 1, characterized in that, The acquisition of multi-level runtime status data of the container orchestration engine includes: Obtain the scheduling determination result of the scheduling component in the container orchestration engine; If the scheduling determination result indicates that the scheduling is successful, the multi-level status monitoring engine is activated and the multi-level running status data of the container orchestration engine is obtained.

3. The state monitoring method for deep learning training tasks according to claim 1, characterized in that, The state verification is performed based on the multi-level operating state data according to a preset state verification order to obtain multiple corresponding state verification results, including: Obtain the Pod layer resource allocation information corresponding to the container orchestration engine, and perform status verification based on the Pod layer resource allocation information to obtain the corresponding first status verification result; The first status verification result is used to characterize whether the Pod resource allocation result meets the preset allocation success conditions.

4. The state monitoring method for deep learning training tasks according to claim 3, characterized in that, Also includes: If the preset allocation success condition is met based on the first status verification result, the running status information of the Pod layer corresponding to the container orchestration engine is obtained, and the status verification is performed based on the running status information of the Pod layer to obtain the corresponding second status verification result. The second status verification result is used to characterize whether there are any abnormalities in the running status of each Pod during its running phase.

5. The state monitoring method for deep learning training tasks according to claim 4, characterized in that, After obtaining the Pod layer runtime status information corresponding to the container orchestration engine, the process further includes: If it is determined that the currently running task requires GPU resources, GPU layer status monitoring data is obtained, and status verification is performed based on the GPU layer status monitoring data to obtain the corresponding third status verification result. The third state verification result is used to characterize whether the actual resource allocation record of the container orchestration engine matches the resource allocation information reported by the hardware allocation management component.

6. The state monitoring method for deep learning training tasks according to claim 1, characterized in that, The process of integrating the multiple state verification results to determine the business state judgment result corresponding to the machine learning training task includes: If all the status verification results indicate successful verification, the business status determination result is determined to be normal operation; If any one of the multiple status verification results indicates a verification failure, the business status determination result is determined to be abnormal.

7. The state monitoring method for deep learning training tasks according to claim 1, characterized in that, Also includes: If the business status determination result is determined to be abnormal, the operation abnormality location information is determined based on each of the status verification results.

8. The state monitoring method for deep learning training tasks according to claim 2, characterized in that, Also includes: If the scheduling determination result indicates a scheduling failure, the task scheduling circuit breaker mechanism is triggered, and the scheduling layer anomaly location information is output.

9. A state monitoring device for a deep learning training task, characterized in that, include: The status monitoring module is used to obtain multi-level runtime status data of the container orchestration engine after submitting the machine learning training task to the container orchestration engine; The status verification module is used to perform status verification based on the multi-level operating status data according to a preset status verification order, and obtain multiple corresponding status verification results. The aggregation and determination module is used to integrate the multiple state verification results to determine the business state determination result corresponding to the machine learning training task.

10. The state monitoring device for a deep learning training task according to claim 9, characterized in that, The status monitoring module is specifically used for: Obtain the scheduling determination result of the scheduling component in the container orchestration engine; If the scheduling determination result indicates that the scheduling is successful, the multi-level status monitoring engine is activated and the multi-level running status data of the container orchestration engine is obtained.

11. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the state monitoring method for the deep learning training task according to any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the state monitoring method for a deep learning training task as described in any one of claims 1-8.

Images