Task detection method and device, electronic equipment and storage medium

Abstract

The application provides a task detection method and device, electronic equipment and storage medium. The method comprises: obtaining task characteristics and constraint conditions of a training task; generating a plurality of detection items and execution order relationships between the detection items according to the task characteristics and the constraint conditions; determining a first detection item and a second detection item from the plurality of detection items according to the execution order relationships between the detection items, wherein the first detection item is a detection item without a preceding dependency relationship determined according to the execution order relationships, and the second detection item is a detection item with a preceding dependency relationship determined according to the execution order relationships; executing the first detection items in parallel, and executing the second detection items in sequence according to the preceding dependency relationships of the second detection items. Thus, the overall detection time is significantly shortened under the premise of ensuring correct detection logic.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and distributed computing technology, and in particular to a task detection method, apparatus, electronic device and storage medium. Background Technology

[0002] Distributed training is a core technology for training current artificial intelligence models. By splitting the training task across multiple computing nodes for parallel execution, the training time can be significantly shortened. Before starting a distributed training task, it is usually necessary to perform health checks on the hardware status, software environment, and communication links of the cluster computing nodes to eliminate potential faults and avoid training tasks failing midway due to computing node failures, thus wasting computing resources and time.

[0003] An existing pre-training computing power testing scheme is as follows: a fixed list of testing items is preset, covering hardware basic computing power testing, video memory subsystem testing, kernel scheduling performance testing, inter-node full topology communication testing, system environment and driver configuration consistency verification, training scenario simulation testing, etc. All testing items are executed serially in a fixed order, adapting to all training task scenarios without difference.

[0004] However, in the above-mentioned existing technical solutions, all detection items are executed sequentially according to a preset order, which causes the overall detection time to increase linearly with the number of detection items, resulting in excessively long detection time. Summary of the Invention

[0005] This application provides a task detection method, apparatus, electronic device, and storage medium to solve the problem in the prior art where all detection items are executed sequentially according to a preset order, resulting in an overall detection time that increases linearly with the number of detection items and an excessively long detection duration.

[0006] Firstly, this application provides a task detection method, including: Obtain the task characteristics and constraints of the training task; Based on the task characteristics and the constraints, multiple detection items and the execution order relationship between the detection items are generated. Based on the execution order relationship between the various detection items, a first detection item and a second detection item are determined from the plurality of detection items, wherein the first detection item is a detection item without a prerequisite dependency determined according to the execution order relationship, and the second detection item is a detection item with a prerequisite dependency determined according to the execution order relationship. Each of the first detection items is executed in parallel, and each of the second detection items is executed sequentially according to their prerequisite dependencies.

[0007] In one possible implementation, generating multiple detection items and the execution order relationship between the detection items based on the task characteristics and the constraints includes: The task features and the constraints are input into a pre-trained reinforcement learning model, wherein the reinforcement learning model is pre-trained with the longest detection time and fault tolerance as constraints, and the optimization objectives being to maximize detection coverage and minimize detection time. Obtain the multiple detection items output by the reinforcement learning model and the execution order relationship between the detection items.

[0008] In one possible implementation, the method further includes: After all test items have been completed, collect the test operation data for this test process; After starting the training task, collect the operational stability data of the training task during its execution. The reinforcement learning model is retrained using the operational stability data and the detection operational data.

[0009] In one possible implementation, the method further includes: During the execution of each of the aforementioned detection items, real-time detection results for each of the aforementioned detection items are collected; When any real-time detection result indicates the existence of a preset fault, all unexecuted detection items are terminated, and a detection failure result is output. The preset fault is an unrecoverable fault. When the real-time detection result indicates the existence of a faulty node and the number of faulty nodes does not exceed the preset fault tolerance threshold, the execution of the detection item that needs to be executed on the faulty node is cancelled, and other detection items are executed. When the number of faulty nodes exceeds the preset fault tolerance threshold, all unexecuted detection items are terminated, and a detection failure result is output.

[0010] In one possible implementation, the method further includes: Before executing each detection item, the first historical detection result of the training task is queried, and the first historical detection result includes the first sub-detection result of each detection item. When the training task and the historical training task corresponding to the first historical detection result meet the reuse condition, the state corresponding to each detection item is determined. The state is used to characterize whether the node hardware configuration and software environment on which the detection item depends have changed. For the third detection item whose state has not changed, query the first sub-detection result corresponding to the third detection item from the first historical detection result, and use the first sub-detection result as the detection result of the third detection item in this detection. For the fourth detection item whose state has changed, execute the fourth detection item to obtain the detection result of the fourth detection item in this detection.

[0011] In one possible implementation, the method further includes: Before executing each detection item, the second historical detection result of the training task is queried. The second historical detection result includes the second sub-detection result of each detection item and the corresponding storage time. When the training task and the historical training task corresponding to the second historical detection result meet the reuse condition, obtain the validity period threshold corresponding to each detection item. The storage duration of the second sub-detection result of each detection item is compared with the corresponding validity period threshold; For the fifth detection item whose storage duration has not exceeded the validity period threshold, the second sub-detection result of the fifth detection item is reused as the detection result of this detection. For the sixth detection item where the storage duration exceeds the validity period threshold, the sixth detection item is executed.

[0012] In one possible implementation, the method further includes: During the execution of each of the aforementioned detection items, a timeout threshold corresponding to each of the aforementioned detection items is obtained, wherein the timeout threshold is dynamically allocated based on the longest detection duration; When the execution time of any detection item exceeds the corresponding timeout threshold, the execution of the detection item is terminated and a warning sign is generated, and the execution of other detection items continues; When the difference between the longest detection duration in the constraints and the total detection duration already executed is less than a preset duration threshold, detection items with a priority lower than the preset priority threshold are removed from the remaining detection items.

[0013] Secondly, this application provides a task detection device, comprising: The acquisition module is used to acquire the task features and constraints of the training task; The generation module is used to generate multiple detection items and the execution order relationship between the detection items based on the task characteristics and the constraints. The determining module is configured to determine a first detection item and a second detection item from a plurality of detection items based on the execution order relationship between the detection items, wherein the first detection item is a detection item without a prerequisite dependency determined based on the execution order relationship, and the second detection item is a detection item with a prerequisite dependency determined based on the execution order relationship. An execution module is configured to execute each of the first detection items in parallel, and to execute each of the second detection items sequentially according to their prerequisite dependencies.

[0014] In one possible implementation, the generation module is specifically used for: The task features and constraints are input into a pre-trained reinforcement learning model, wherein the reinforcement learning model is pre-trained with the longest detection time and fault tolerance as constraints, and the optimization objectives being to maximize detection coverage and minimize detection time. Obtain the multiple detection items output by the reinforcement learning model and the execution order relationship between the detection items.

[0015] In one possible implementation, the device further includes a training module for: After all test items have been completed, collect the test operation data for this test process; After starting the training task, collect the operational stability data of the training task during its execution. The reinforcement learning model is retrained using the operational stability data and the detection operational data.

[0016] In one possible implementation, the device further includes a stop module for: During the execution of each of the aforementioned detection items, real-time detection results for each of the aforementioned detection items are collected; When any real-time detection result indicates the existence of a preset fault, all unexecuted detection items are terminated, and a detection failure result is output. The preset fault is an unrecoverable fault. When the real-time detection result indicates the existence of a faulty node and the number of faulty nodes does not exceed the preset fault tolerance threshold, the execution of the detection item that needs to be executed on the faulty node is cancelled, and other detection items are executed. When the number of faulty nodes exceeds the preset fault tolerance threshold, all unexecuted detection items are terminated, and a detection failure result is output.

[0017] In one possible implementation, the device further includes a first multiplexing module for: Before executing each detection item, the first historical detection result of the training task is queried, and the first historical detection result includes the first sub-detection result of each detection item. When the training task and the historical training task corresponding to the first historical detection result meet the reuse condition, the state corresponding to each detection item is determined. The state is used to characterize whether the node hardware configuration and software environment on which the detection item depends have changed. For the third detection item whose state has not changed, query the first sub-detection result corresponding to the third detection item from the first historical detection result, and use the first sub-detection result as the detection result of the third detection item in this detection. For the fourth detection item whose state has changed, execute the fourth detection item to obtain the detection result of the fourth detection item in this detection.

[0018] In one possible implementation, the device further includes a second multiplexing module for: Before executing each detection item, the second historical detection result of the training task is queried. The second historical detection result includes the second sub-detection result of each detection item and the corresponding storage time. When the training task and the historical training task corresponding to the second historical detection result meet the reuse condition, obtain the validity period threshold corresponding to each detection item. The storage duration of the second sub-detection result of each detection item is compared with the corresponding validity period threshold; For the fifth detection item whose storage duration has not exceeded the validity period threshold, the second sub-detection result of the fifth detection item is reused as the detection result of this detection. For the sixth detection item where the storage duration exceeds the validity period threshold, the sixth detection item is executed.

[0019] In one possible implementation, the device further includes a removal module for: During the execution of each of the aforementioned detection items, a timeout threshold corresponding to each of the aforementioned detection items is obtained, wherein the timeout threshold is dynamically allocated based on the longest detection duration; When the execution time of any detection item exceeds the corresponding timeout threshold, the execution of the detection item is terminated and a warning sign is generated, and the execution of other detection items continues; When the difference between the longest detection duration in the constraints and the total detection duration already executed is less than a preset duration threshold, detection items with a priority lower than the preset priority threshold are removed from the remaining detection items.

[0020] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a task detection program stored in the memory to implement the task detection method described in any one of the first aspects.

[0021] Fourthly, this application provides a storage medium storing one or more programs that can be executed by one or more processors to implement the task detection method described in any one aspect.

[0022] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application first obtains the task features and constraints of the training task, and generates multiple detection items and the execution order relationship between each detection item accordingly; based on this, a first detection item without a prerequisite dependency and a second detection item with a prerequisite dependency are determined from the multiple detection items according to the execution order relationship; then, the first detection item without a prerequisite dependency is executed in parallel, and the second detection item with a prerequisite dependency is executed sequentially according to its prerequisite dependency. Through the above method, this application can achieve differentiated scheduling based on the prerequisite dependency relationship between detection items, so that detection items without a prerequisite dependency relationship can be executed simultaneously, avoiding the linear increase in time consumption caused by the forced serial execution regardless of whether a dependency relationship exists in the prior art, thereby significantly shortening the overall detection time while ensuring the correctness of the detection logic. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0026] Figure 1 A flowchart illustrating an embodiment of a task detection method provided in this application; Figure 2 A flowchart illustrating another embodiment of the task detection method provided in this application; Figure 3 A flowchart illustrating another embodiment of the task detection method provided in this application; Figure 4 A block diagram illustrating an embodiment of a task detection device provided in this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0029] To address the technical problem in existing technologies where all detection items are executed sequentially according to a preset order, resulting in a linear increase in overall detection time with the number of detection items and excessively long detection duration, this application provides a task detection method that can achieve differentiated scheduling based on the pre-dependencies between detection items. This allows detection items without pre-dependencies to be executed simultaneously, avoiding the linear increase in time caused by the forced sequential execution regardless of whether dependencies exist in existing technologies. Thus, the overall detection time is significantly shortened while ensuring the correctness of the detection logic.

[0030] Figure 1 This is a flowchart illustrating an embodiment of a task detection method provided in this application. Figure 1 As shown, the method includes the following steps: Step 101: Obtain the task characteristics and constraints of the training task.

[0031] Training task: refers to the distributed deep learning training task to be performed, such as a large language model training task or a computer vision model training task.

[0032] Task characteristics: refers to the core attribute information of the training task to be detected, including model type, distributed training mode, number of instances (i.e., number of nodes), number of GPUs (Graphics Processing Units) per node, and GPU card specifications.

[0033] Constraints: These refer to the service quality assurance requirements and limiting parameters of the training task, including the maximum detection duration set by the user, the task failure tolerance, the SLA (Service Level Agreement) level, and the expected task operation cycle.

[0034] In this embodiment, step 101 obtains the aforementioned task characteristics and constraints by automatically parsing the training task configuration file submitted by the user or by interacting with a cluster scheduling system (such as Kubernetes or Slurm). For example, when a user submits a 32-node, 20B large language model training task, step 101 automatically parses the task characteristics as follows: model type is Transformer, distributed training mode is 3D parallel (including three core parallel dimensions: data parallelism, tensor parallelism, and pipeline parallelism), number of instances is 32 nodes, number of GPUs per node is 8, and GPU card type is A800; at the same time, the constraints are parsed as follows: the maximum detection time set by the user is 30 minutes, the fault tolerance is extremely low (the number of faulty nodes allowed is 0), the SLA level is the highest, and the expected task running cycle is 30 days. For example, when a user submits a single-node computer vision model training task, step 101 automatically parses the task features as follows: model type is CNN (Convolutional Neural Network), distributed training mode is single-node non-distributed, number of instances is 1 node, number of GPUs per node is 1, and GPU card type is V100; constraints are: maximum detection time is 5 minutes, fault tolerance is medium (allowing 1 faulty node), SLA level is normal, and the expected task execution cycle is 2 hours. These task features and constraints will serve as input for subsequent steps to generate the detection strategy.

[0035] Step 102: Based on the task characteristics and the constraints, generate multiple detection items and the execution order relationship between each detection item.

[0036] Testing items: These are independent test units that perform health checks on the hardware status, software environment, or communication links of computing nodes. Their content can be dynamically determined according to the task scenario. For example, they may include hardware basic computing power testing, video memory subsystem testing, memory subsystem testing, kernel scheduling performance testing, inter-node communication testing, system environment consistency verification, scenario simulation testing, etc.

[0037] Execution order relationship: refers to the dependency constraints between each detection item in terms of execution time, used to determine the execution order of detection items and the detection items that can be executed in parallel.

[0038] In this embodiment, step 102 generates a combination of detection items adapted to the current task scenario based on the task features and constraints obtained in step 101, and clarifies the execution order relationship between each detection item. Specifically, step 102 can use various strategies to generate detection items and execution order, such as based on preset rules, based on reinforcement learning models (e.g., DQN, Deep Q-Network), based on heuristic rule engines, based on genetic algorithms, based on PPO (Proximal Policy Optimization) deep reinforcement learning models, or based on multi-objective optimization algorithms. The generated combination of detection items can be dynamically adjusted according to the task scale: for small-scale tasks with a single node and a single GPU, distributed communication detection items are automatically skipped, low-priority deep detection items are pruned, and only core detection items such as basic computing power detection, memory fault detection, driver environment consistency verification, and kernel startup detection are retained; for large-scale tasks (e.g., a long-cycle training task with a large model on 32 nodes), all-dimensional detection items are automatically enabled, including basic computing power detection, memory subsystem detection, kernel scheduling detection, full-topology protocol-level communication detection, system configuration verification, and scenario simulation detection. There are inherent dependencies between different detection items. For example, full-topology communication detection can only be executed after the system configuration verification is completed and passed, because communication detection depends on the correct network driver and protocol configuration; scenario simulation detection can only be started after all hardware detection items are completed, because simulation detection requires a healthy hardware environment; there is no dependency between memory fault detection and kernel scheduling performance detection, and they can be executed in any order. Step 102 generates the execution order relationship between each detection item based on these inherent dependencies, and sorts them according to the priority of the detection items (preset according to the degree to which the fault affects the success or failure of training, with memory fault detection having the highest priority of P0, computing power performance benchmark detection having the next highest priority of P1, and full simulation detection of training scenarios having the lowest priority of P2), providing a basis for subsequent parallel scheduling.

[0039] Step 103: Based on the execution order relationship between the detection items, determine the first detection item and the second detection item from the plurality of detection items, wherein the first detection item is a detection item without a prerequisite dependency determined according to the execution order relationship, and the second detection item is a detection item with a prerequisite dependency determined according to the execution order relationship.

[0040] Pre-dependency: refers to the execution timing constraints between detection items, that is, one detection item can only start execution after another detection item has been executed and passed.

[0041] The first detection item refers to the detection item that has no prior dependencies. That is, these detection items can be executed independently without depending on the execution results of other detection items and can be scheduled in parallel.

[0042] The second detection item refers to a detection item that has at least one prerequisite dependency. That is, these detection items can only be executed after the prerequisite detection items they depend on have been executed and passed. They need to be scheduled sequentially.

[0043] In this embodiment, step 103 involves parsing the execution order relationship generated in step 102 to construct a detection execution DAG (Directed Acyclic Graph). Specifically, each independent detection item is treated as a node in the DAG; the pre-dependencies between detection items are analyzed, and directed edges are added between nodes with dependencies; detection items without any pre-dependencies, i.e., nodes with an in-degree of zero in the DAG, are marked as first detection items (nodes that can be executed in parallel); detection items with pre-dependencies, i.e., nodes with incoming edges in the DAG, are marked as second detection items (nodes that are executed sequentially).

[0044] In another implementation, a dependency matrix or task flow topology table can be used to identify and schedule dependency-free detection items in parallel. This eliminates the need to construct a complete DAG graph; parallel execution strategies can be generated simply by matching dependencies. For example, in a 32-node large model training task, there are no mutual dependencies between basic computing power detection, memory subsystem detection, kernel scheduling detection, and system configuration verification, i.e., no prerequisite dependencies, so they are all identified as the first detection item. Full-topology communication detection depends on the completion of system configuration verification, i.e., there is a prerequisite dependency (system configuration verification is its prerequisite item), so it is identified as the second detection item. Scene simulation detection depends on the completion of all hardware detection items, including basic computing power detection, memory subsystem detection, kernel scheduling detection, system configuration verification, and full-topology communication detection, and has multiple prerequisite dependencies, so it is also identified as the second detection item. Similarly, in a single-node CNN (Convolutional Neural Network) small model training task, there are no prerequisite dependencies between basic computing power detection, memory fault detection, driver environment consistency verification, and kernel startup detection, so they are all identified as the first detection item.

[0045] Step 104: Execute each of the first detection items in parallel, and execute each of the second detection items sequentially according to their prerequisite dependencies.

[0046] Parallel execution: This refers to starting multiple detection items simultaneously using a multi-threaded or multi-process scheduler, allowing them to run concurrently on computing resources to reduce detection time. In another implementation, multi-process parallel or distributed multi-node parallel scheduling can also be used to distribute different detection items to different nodes for parallel execution for detection tasks on very large clusters.

[0047] Sequential execution: This means executing the detection items one by one in the order determined by the preceding dependencies. Only after the preceding dependency has been executed and passed the test will the subsequent related detection items be started.

[0048] In this embodiment, step 104 first starts the scheduler and simultaneously starts all the detection items marked as the first detection item without prerequisite dependencies, so that they are executed in parallel at the same time. For the detection items marked as the second detection items with prerequisite dependencies, the subsequent detection items are started in sequence according to the execution order determined in step 102, after the prerequisite items have been executed and passed the detection.

[0049] The parallel granularity of the scheduler can be dynamically configured based on available computing resources. For example, a multi-threaded scheduler can be launched, and the number of threads can be dynamically adjusted according to the node's computing power (e.g., 4 threads, 8 threads, or 16 threads). Specifically, in the detection of the 32-node large model training task, step 104 launches an 8-thread scheduler and executes four first detection items in parallel: basic computing power detection, memory subsystem detection, kernel scheduling performance detection, and system configuration verification. These four detection items run concurrently, taking approximately 2 minutes to complete (while serial execution would require accumulating the time of each detection item, approximately 2 + 2 + 2 + 2 = 8 minutes). Subsequently, after the system configuration verification detection passes, the full topology communication detection (second detection item) is automatically launched, taking approximately 8 minutes to complete. Finally, after all hardware detection items (including basic computing power detection, memory subsystem detection, kernel scheduling detection, system configuration verification, and full topology communication detection) pass, the scene simulation detection (second detection item) is automatically launched, taking approximately 1 minute to complete. In the single-node CNN small model training task detection, step 104 starts a 4-thread scheduler to execute all four first detection items (basic computing power detection, memory fault detection, driver environment consistency verification, and kernel startup detection) in parallel, which takes about 22 seconds in total.

[0050] Furthermore, during parallel scheduling, if a prerequisite item fails to execute, all subsequent associated detection items for that link are immediately terminated, and the link is marked as failed to avoid wasted time. For example, if system configuration verification fails, the full-topology communication detection and scenario simulation detection that depend on it will no longer be executed.

[0051] The technical solution provided in this application first obtains the task features and constraints of the training task, and generates multiple detection items and their execution order relationships based on these features. Then, according to the execution order relationship, a first detection item without pre-dependencies and a second detection item with pre-dependencies are determined from the multiple detection items. Next, the first detection item without pre-dependencies is executed in parallel, while the second detection item with pre-dependencies is executed sequentially according to its pre-dependency. Through this method, this application can achieve differentiated scheduling based on the pre-dependencies between detection items, enabling detection items without pre-dependencies to be executed simultaneously. This avoids the linear increase in time consumption caused by the forced serial execution regardless of whether a dependency exists in the prior art, thereby significantly shortening the overall detection time while ensuring the correctness of the detection logic.

[0052] In one embodiment, step 102 may specifically include the following steps: inputting the task features and the constraints into a pre-trained reinforcement learning model, wherein the reinforcement learning model is pre-trained with the longest detection time and fault tolerance as constraints and the optimization objectives of maximizing detection coverage and minimizing detection time; and obtaining multiple detection items output by the reinforcement learning model and the execution order relationship between each detection item.

[0053] Reinforcement learning model: refers to a decision model trained based on reinforcement learning algorithm. In this embodiment, DQN (Deep Q-Network) is used. This model can output the optimal decision action (i.e., the combination of detection items and the order of execution) based on the input state information (i.e., task features and constraints).

[0054] Maximum detection duration: refers to the maximum allowed detection time set by the user for the training task, used to constrain the upper limit of the total duration of the detection process.

[0055] Fault tolerance: refers to the training task's ability to withstand node failures, such as the maximum number of nodes that can fail.

[0056] Detection coverage: refers to the proportion of the generated combination of detection items relative to the full set of detection items, or the coverage of potential fault types that can be detected.

[0057] Detection time: refers to the total time required to execute the generated combination of detection items.

[0058] In this embodiment, step 102 specifically uses a pre-trained reinforcement learning model to generate the relationship between detection items and execution order. First, the task features obtained in step 101 (such as model type, distributed training mode, number of nodes, number of GPUs per node, GPU card specifications, etc.) and constraints (such as maximum detection time, fault tolerance, SLA level, expected task execution cycle, etc.) are passed as input states to the pre-trained DQN model. This DQN model uses the maximum detection time and fault tolerance as hard constraints to ensure that the output detection strategy does not exceed the user-defined time limit and meets the task's fault tolerance requirements; at the same time, with the dual optimization objectives of maximizing detection coverage and minimizing detection time, the optimal combination of detection items, the execution order of each detection item, and the timeout threshold of a single detection item are calculated through the policy network inside the model. Specifically, the reward function of the DQN model is designed as a weighted sum of positive rewards for coverage and negative penalties for time consumption, i.e., Reward = Coverage × α - (Time Consumption / Longest Detection Time) × β, where α and β are weight coefficients. By adjusting the ratio of α and β, the importance of coverage and time consumption can be balanced. At the model output, the optimal combination of detection items generated by the DQN model and the execution order relationship between each detection item are obtained.

[0059] Based on the detection strategy output by this model, scenario-based adaptive optimization is achieved: for single-node training tasks, distributed communication-related detection items are automatically skipped; for short-cycle, low-SLA training tasks, low-priority detection items are automatically pruned, and core fault detection items are prioritized; for long-cycle, high-SLA large-model training tasks, full-dimensional deep detection items are automatically enabled to ensure full fault coverage. The priority of the detection items is statically set, with fixed priorities preset according to the degree to which faults affect the success or failure of training. For example, memory / GPU fault detection has the highest priority (P0), computing power performance benchmark detection has the next highest priority (P1), and full-scale simulation detection of training scenarios has the lowest priority (P2).

[0060] This solution adaptively generates detection strategies through a pre-trained reinforcement learning model, which can dynamically adjust the combination of detection items and the execution order according to the characteristics and constraints of different tasks. This avoids the shortcomings of existing technologies where fixed detection processes cannot adapt to different task scenarios. Under the premise of meeting the constraints of the longest detection time and fault tolerance, it achieves the optimal trade-off between detection coverage and detection time, which not only ensures the comprehensiveness of fault detection for long-cycle, large-model tasks, but also significantly reduces the detection time for short-cycle, small-model tasks.

[0061] In another embodiment, the method may further include the following steps: after all detection items have been executed, collecting detection operation data of this detection process; after starting the training task, collecting operation stability data of the training task during operation; and retraining the reinforcement learning model using the operation stability data and the detection operation data.

[0062] Test operation data: refers to the data related to the test process collected during the execution of each test item, including but not limited to the actual time consumed by each test item, the fault detection status, the test coverage, etc.

[0063] Operational stability data refers to data related to task stability collected during the operation of the training task after it starts, including but not limited to hardware failure indicators (such as the number of GPU XID errors, the number of GPU downclocking times, and the number of memory errors), communication failure indicators (such as the number of inter-node communication timeouts, the number of 3D parallel synchronization failures, and the number of link disconnections), and training convergence indicators (such as whether the loss is diverging and the training throughput jitter rate).

[0064] Retraining: refers to iteratively updating the reinforcement learning model using newly collected experience samples to optimize the detection strategies generated by the model in the future.

[0065] In this embodiment, firstly, after all detection items are executed, the detection operation data generated during this detection process is collected, including the actual time consumed by each detection item, fault detection status, detection coverage, etc. These data reflect the execution effect of this detection strategy. Secondly, after the training task starts, the operational stability data of the training task during its operation is collected. A high-fault observation window is set according to the task cycle: for short-cycle tasks with an expected runtime of less than or equal to 4 hours, the observation window is set to the first 30 minutes after training starts; for long-cycle or large-model tasks with an expected runtime of more than 4 hours, the observation window is set to the first hour after training starts. Within this observation window, the aforementioned hardware fault indicators, communication fault indicators, and training convergence indicators are collected as operational stability data. Then, using the collected detection operation data and operational stability data, the reinforcement learning model is retrained.

[0066] Specifically, a composite update mechanism of basic immediate reward plus delayed stability correction reward is adopted: After detection is completed, the basic reward is calculated based on the detection coverage and detection time (for example, the higher the coverage, the higher the reward, and the longer the time, the greater the penalty); after the observation window ends, the correction reward ΔR is calculated based on the operational stability data (for example, if no hardware failure or communication failure occurs within the observation window and the training converges well, a positive correction reward is given; if a failure occurs or the loss diverges, a negative correction reward is given); finally, the network parameters of the DQN model are updated with the total reward equal to the basic reward plus the correction reward (i.e., total reward = basic reward + ΔR), so that the model pursues high efficiency in the detection stage while taking into account the long-term stability in the training stage.

[0067] This solution uses collected detection operation data and post-training stability data as experience samples, and employs a composite update mechanism of basic reward plus delay stability correction reward to iteratively optimize the reinforcement learning model. This allows the model to continuously learn from actual detection performance and subsequent training stability, constantly improving the rationality of detection strategy planning. This achieves the self-learning and self-optimization capabilities of the detection strategy, making the model "more efficient and more accurate with use," further enhancing the adaptability of the detection process to different task scenarios.

[0068] In another embodiment, the method may further include the following steps: during the execution of each of the detection items, collecting real-time detection results of each of the detection items; when any real-time detection result indicates the existence of a preset fault, terminating all unexecuted detection items and outputting a detection failure result, wherein the preset fault is an unrecoverable fault; when the real-time detection result indicates the existence of a fault node and the number of fault nodes does not exceed a preset fault tolerance threshold, canceling the execution of the detection item that needs to be executed on the fault node and continuing to execute other detection items; when the number of fault nodes exceeds the preset fault tolerance threshold, terminating all unexecuted detection items and outputting a detection failure result.

[0069] Real-time detection results refer to the detection status information obtained in real time during the execution of each detection item, including detection passed, detection failed, detection timed out, etc., as well as the specific fault type and fault node information identified during the detection process.

[0070] Preset faults: refer to predefined fault types that need to trigger rapid termination. In this embodiment, the preset faults are unrecoverable serious faults, including but not limited to hardware faults (such as GPU card drop, uncorrectable memory errors), configuration inconsistencies (such as GPU driver version and CUDA version mismatch, container image loading failure), communication link interruption (such as RDMA connection between nodes disconnection, network card failure), etc.

[0071] Fault node: refers to a computing node that is identified as having the above-mentioned preset faults or other abnormalities during the execution of the detection items.

[0072] Preset fault tolerance threshold: This refers to the maximum number of faulty nodes that the user is allowed to occur. If this number is exceeded, the detection is considered a failure, and the stable operation of the training task cannot be guaranteed.

[0073] In this embodiment, during the execution of each detection item, the detection results of each detection item are collected in real time, including detection status, fault type, fault node identifier, and other information. When any real-time detection result indicates the existence of a preset fault (i.e., an unrecoverable severe fault), a fast termination mechanism is immediately triggered, terminating all unexecuted detection items and directly outputting the detection failure result without waiting for the full detection to complete, thereby maximizing the reduction of invalid detection time. When the real-time detection result indicates the existence of a fault node, but the number of fault nodes does not exceed the user-preset fault tolerance threshold, a fault node isolation strategy is adopted: canceling the execution of all detection items that need to be executed on the fault node (i.e., skipping subsequent detections that depend on the fault node), and continuing to execute other unrelated detection items, ensuring that the detection process can obtain detection information from as many healthy nodes as possible. When the number of fault nodes exceeds the preset fault tolerance threshold, it is determined that the current cluster state cannot meet the minimum health requirements of the training task, triggering a fast termination mechanism, terminating all unexecuted detection items, and outputting the detection failure result, along with fault details and fault node location information, to facilitate user troubleshooting and node replacement.

[0074] This solution identifies serious faults in real time and immediately terminates the full detection process, stopping invalid detections as soon as an unrecoverable fault is detected, thus minimizing detection time. At the same time, through a fault node tolerance and isolation mechanism, fault nodes are automatically isolated and other detection items continue to be executed when the number of fault nodes does not exceed the tolerance threshold. This avoids the unnecessary overhead of interrupting the full detection due to a single node failure, maximizing detection efficiency and node utilization while ensuring the basic operating conditions of the training task.

[0075] Figure 2 A flowchart illustrating another embodiment of the task detection method provided in this application. Figure 2 The process shown is in Figure 1 Based on the illustrated process, the following steps are included: Step 201: Before executing each detection item, query the first historical detection result of the training task, which includes the first sub-detection result of each detection item; Step 202: When the training task and the historical training task corresponding to the first historical detection result meet the reuse condition, determine the state corresponding to each detection item. The state is used to characterize whether the node hardware configuration and software environment on which the detection item depends have changed. Step 203: For the third detection item whose state has not changed, query the first sub-detection result corresponding to the third detection item from the first historical detection result, and use the first sub-detection result as the detection result of the third detection item in this detection. For the fourth detection item whose state has changed, execute the fourth detection item to obtain the detection result of the fourth detection item in this detection.

[0076] For ease of understanding, steps 201-203 will be explained uniformly below: First historical test results: refers to the historical test records generated when the same training task or the same node group is executed, including the historical sub-test results of each test item (i.e. the pass / fail status and detailed test data of each test item in the previous test).

[0077] Reuse conditions: These refer to the prerequisites for allowing the reuse of historical detection results, including the same set of physical nodes corresponding to the training task and the historical training task (i.e., the set of unique hardware identifiers of the nodes is completely consistent) and the network topology between the nodes has not changed.

[0078] Status: refers to whether the hardware configuration and software environment of the node on which the detection item depends have changed. It is determined by status fingerprint verification and includes two dimensions: hardware status and software environment status.

[0079] The third detection item refers to the detection item whose status has not changed, that is, the node hardware configuration and software environment on which the detection item depends are completely consistent with those in the historical detection.

[0080] The fourth detection item refers to the detection item whose status has changed, that is, the node hardware configuration or software environment on which the detection item depends has changed compared with the previous detection.

[0081] In this embodiment, step 201: Before executing each detection item, first query the first historical detection result of the training task. The first historical detection result includes the first sub-detection result of each detection item in the last detection (such as whether each detection item passed, the specific detection value, etc.). Step 202: Determine whether the current training task and the historical training task corresponding to the first historical detection result meet the reuse condition, that is, whether the physical node sets corresponding to the two are completely the same (including hardware unique identifiers such as node SN code, GPU UUID, and network card MAC address), and whether the network topology between the nodes has not changed. When the reuse condition is met, further determine the state corresponding to each detection item. This state is used to characterize whether the node hardware configuration and software environment on which the detection item depends have changed. Specifically, the state is determined through state fingerprint verification: a hardware hash value is generated from the unique identifier of the node hardware (including GPU UUID, card type, memory capacity, node SN code, network card MAC address, CPUID, etc.), and an environment hash value is generated from the software configuration (including GPU driver version, CUDA version, cuDNN version, system kernel version, container image ID, deep learning framework version, key environment variables, etc.). The current hardware hash value and environment hash value are compared with the hash values ​​saved during historical detection. If the hash values ​​of the two detections are consistent, it is determined that the node hardware has not been replaced or faulted, the software environment has not been modified, and the state has not changed; otherwise, it is determined that the state has changed. Step 203: For the third detection item whose state has not changed, the first sub-detection result corresponding to the detection item is directly queried from the first historical detection result, and the historical result is used as the detection result of this detection, without re-executing the detection; for the fourth detection item whose state has changed, the detection item is executed to obtain the real-time detection result of this detection.

[0082] pass Figure 2 The process shown can automatically reuse historical detection results within the validity period for training tasks that are restarted or repeatedly submitted within the same node group, and only perform incremental change detection items. This avoids full duplicate detection for scenarios involving task restarts or repeated tasks within the same node group, reducing the detection time from several minutes or more for full execution to seconds (within 10 seconds), while also avoiding the waste of computing resources caused by duplicate detection.

[0083] Figure 3 A flowchart illustrating another embodiment of the task detection method provided in this application. Figure 3 The process shown is in Figure 1 Based on the illustrated process, the following steps are included: Step 301: Before executing each detection item, query the second historical detection result of the training task. The second historical detection result includes the second sub-detection result of each detection item and the corresponding storage time. Step 302: When the training task and the historical training task corresponding to the second historical detection result meet the reuse condition, obtain the validity period threshold corresponding to each detection item; Step 303: Compare the storage duration of the second sub-detection result of each detection item with the corresponding validity period threshold; Step 304: For the fifth detection item whose storage duration has not exceeded the validity period threshold, reuse the second sub-detection result of the fifth detection item as the detection result of this detection; for the sixth detection item whose storage duration has exceeded the validity period threshold, execute the sixth detection item.

[0084] For ease of understanding, steps 301-304 are explained uniformly below: Second historical detection results: refers to the historical detection records generated when the same training task or the same node group is executed. It includes the historical sub-detection results of each detection item (i.e., the pass / fail status and detailed detection data of the last detection of each detection item) and the storage duration of the detection result (i.e. the time elapsed from the completion of the detection to the current moment).

[0085] Storage duration: refers to the length of time elapsed from the generation of the detection result to the current moment, used to determine whether the detection result is still within its validity period.

[0086] Reuse conditions: These refer to the prerequisites for allowing the reuse of historical detection results, including the same set of physical nodes corresponding to the training task and the historical training task (i.e., the set of unique hardware identifiers of the nodes is completely consistent) and the network topology between the nodes has not changed.

[0087] Validity period threshold: refers to the validity period of the test results set for different test items. Different types of test items have different validity period thresholds. For example, the validity period threshold for basic hardware test can be set to 24 hours, the validity period threshold for environmental configuration test can be set to 2 hours, and the validity period threshold for communication test can be set to 30 minutes.

[0088] The fifth detection item refers to the detection item whose storage time has not exceeded the corresponding validity period threshold. That is, the historical detection results of this detection item are still within the validity period and can be directly reused.

[0089] The sixth detection item refers to the detection item whose storage time exceeds the corresponding validity period threshold. That is, the historical detection results of this detection item have expired and need to be re-executed.

[0090] In this embodiment, step 301: Before executing each detection item, first query the second historical detection result of the training task. The second historical detection result includes not only the second sub-detection result of each detection item (such as whether each detection item passed, specific detection values, etc.), but also the storage duration corresponding to each detection result (i.e., the time interval from the completion of the detection to the current moment). Step 302: Determine whether the current training task and the historical training task corresponding to the second historical detection result meet the reuse condition, that is, whether the physical node sets corresponding to the two are exactly the same and whether the network topology has not changed. When the reuse condition is met, further obtain the validity period threshold corresponding to each detection item. It should be noted that the validity period thresholds for different types of detection items can be set differently. For example, the validity period threshold for basic hardware detection (such as whether there are potential hardware errors in the GPU) is relatively long and can be set to 24 hours, because the hardware status usually does not change in the short term; the validity period threshold for environment configuration detection (such as GPU driver version, CUDA version, container image ID, etc.) can be set to 2 hours, because the software environment may be modified during task debugging; the validity period threshold for communication detection (such as the RDMA communication link status between nodes) can be set to 30 minutes, because the communication link status is greatly affected by factors such as network load and needs to be detected more frequently. Step 303: Compare the storage duration of the second sub-detection result of each detection item with the validity period threshold corresponding to that detection item to determine whether the storage duration exceeds the validity period threshold. Step 304: For the fifth detection item whose storage duration does not exceed the validity period threshold, directly reuse the second sub-detection result of that detection item as the detection result of this detection, without re-executing the detection; for the sixth detection item whose storage duration exceeds the validity period threshold, re-executing the detection item to obtain the real-time detection result of this detection. By using the above method, hierarchical incremental detection can be achieved based on the validity period of the detection item in scenarios where the conditions for state fingerprint reuse are not met (such as the lack of historical detection results or the need for more refined timeliness control), thereby further reducing the time consumed by repeated detection.

[0091] This mechanism can be used in combination with the state fingerprint reuse mechanism (i.e., the embodiment of steps 201-203 above). For example, first determine whether there has been a change based on the state fingerprint. If there has been no change, then further determine whether the storage duration is within the validity period, thereby providing a multi-level incremental detection strategy.

[0092] Figure 3The process illustrated here, by setting validity period thresholds for different detection items and determining whether to reuse or re-execute based on the comparison between the storage duration of the detection result and the validity period threshold, enables hierarchical incremental reuse of detection results in scenarios where the conditions for state fingerprint reuse are not met or where more granular timeliness control is required. Compared to the state fingerprint reuse mechanism, the validity period reuse mechanism does not require complex hash value comparison calculations, making it more lightweight. The two mechanisms can be used in combination to provide multi-level incremental detection strategies, further avoiding the waste of computing resources and time caused by repeated detections, and compressing the detection time to the second level.

[0093] In one embodiment, the method may further include the following steps: during the execution of each of the detection items, obtaining a timeout threshold corresponding to each of the detection items, wherein the timeout threshold is dynamically allocated based on the longest detection duration; when the execution duration of any detection item exceeds the corresponding timeout threshold, terminating the execution of the detection item and generating a warning flag, and continuing to execute other detection items; when the difference between the longest detection duration in the constraint condition and the total execution duration is less than a preset duration threshold, removing detection items with a priority lower than a preset priority threshold from the remaining detection items.

[0094] Timeout threshold: This refers to the maximum allowed execution time set for each detection item or each detection stage. Exceeding this time limit indicates that the detection item has timed out. In this embodiment, the timeout threshold is dynamically allocated based on the longest detection time set by the user. Different types of detection items may have different timeout thresholds. For example, the timeout threshold for full-topology communication detection, which takes a relatively long time, can be set to 10 minutes, while the timeout threshold for basic computing power detection, which takes a relatively short time, can be set to 2 minutes.

[0095] Execution time: refers to the running time of a detection item from the start of execution to the current moment.

[0096] Warning indicator: This indicates the status information used to mark timeout detection items, indicating that the detection item failed to complete due to timeout, but it does not affect the execution of other unrelated detection items.

[0097] Preset duration threshold: This refers to the remaining time warning value used to trigger the dynamic pruning mechanism. When the difference between the longest detection duration and the total detection duration already executed is less than this threshold, it indicates that there is insufficient remaining time and the pruning strategy needs to be activated.

[0098] Priority: refers to the importance level of the detection item. Fixed priorities are preset according to the degree to which the fault affects the success or failure of training. For example, the highest priority of video memory / memory fault detection is P0, the next highest priority of computing power performance benchmark detection is P1, and the lowest priority of training scenario full simulation detection is P2.

[0099] Preset priority threshold: refers to the priority boundary used to filter the detection items to be pruned. When pruning is required, detection items with a priority lower than this threshold are removed, and the execution of high-priority detection items is guaranteed first.

[0100] In this embodiment, before or during the execution of each detection item, a corresponding timeout threshold is dynamically allocated to each detection item based on the user-set maximum detection duration. The allocation strategy can be differentiated based on the expected time consumption of the detection item and historical execution experience: a larger timeout threshold is allocated for detection items with a longer expected time consumption (such as full-topology communication detection); a smaller timeout threshold is allocated for detection items with a shorter expected time consumption (such as basic computing power detection). Simultaneously, corresponding timeout thresholds are also allocated for different detection stages to ensure that the overall detection process can be completed within the maximum detection duration. During the execution of each detection item, the execution duration of each detection item is monitored in real time. When the execution duration of any detection item exceeds its corresponding timeout threshold, the execution of that detection item is immediately terminated and a warning flag is generated, marking the detection item as timed out. However, this does not affect the normal execution of other unrelated detection items, and the remaining detection items continue to be scheduled for execution. Furthermore, throughout the entire detection process, the total execution time is monitored in real time, and the difference between the maximum detection duration and the total execution time is calculated, i.e., the remaining available detection time. When the difference is less than a preset time threshold (e.g., less than 1 minute remaining), a dynamic pruning mechanism is triggered: from the remaining unexecuted detection items, detection items with a priority lower than a preset priority threshold are identified and removed (e.g., removing full simulation detection of the training scene with a priority of P2). Only core detection items with a priority at or above the threshold (such as detection items at levels P0 and P1) continue to be executed, thus ensuring that the execution of core detection items is completed first within the longest detection time. For the pruned detection items, they can be marked as skipped or warned according to the configuration, but this does not affect the overall decision on the detection results.

[0101] This solution dynamically allocates timeout thresholds for each detection item based on the longest detection duration, and promptly terminates the execution of a detection item when it times out, allowing other detection items to continue executing. This avoids the entire detection process being blocked due to a single detection item being stuck. At the same time, through a dynamic pruning mechanism, low-priority detection items are automatically removed when there is insufficient remaining detection time, prioritizing the execution of core detection items. This ensures that core detections are completed within the user-set maximum detection duration, completely eliminating the risk of task startup failure due to detection timeouts and increasing the user's proactive startup rate.

[0102] To facilitate understanding of the technical solution of this application, the task detection method provided in one embodiment of this application will be described in its entirety below.

[0103] The method comprises two phases: a detection process planning phase and a detection execution phase. In the detection process planning phase, the task characteristics and constraints of the training task are first obtained. Task characteristics include model type (e.g., Transformer), distributed training mode (e.g., data parallelism, tensor parallelism, pipelined parallelism, or 3D parallelism), number of nodes, number of GPUs per node, and GPU card specifications (e.g., A100, H800). Constraints include the user-set maximum detection time (e.g., 30 minutes), task failure tolerance (e.g., maximum allowed number of failed nodes), SLA level, and expected task execution cycle (e.g., short cycle ≤ 4 hours, long cycle > 4 hours). Subsequently, based on the obtained task characteristics and constraints, multiple detection items and their execution order relationships are generated. Specifically, this generation process can be achieved using a pre-trained reinforcement learning model (such as DQN): the task features and constraints are input into the pre-trained DQN model, which uses the longest detection time and fault tolerance as hard constraints, and maximizes detection coverage and minimizes detection time as dual optimization objectives. The model outputs the optimal combination of detection items, the execution order of each detection item, and the timeout threshold of a single detection item through the reward function Reward = coverage × α - (time / longest detection time) × β (where α and β are weight coefficients). The detection strategy based on model output enables scenario-based adaptive adaptation: for single-node training tasks, it automatically skips distributed communication detection items; for short-cycle, low-SLA training tasks, it automatically prunes low-priority detection items (priorities are preset as P0, P1, P2 based on the degree to which faults affect training success or failure), and prioritizes the execution of core fault detection items; for long-cycle, high-SLA large model training tasks, it automatically enables full-dimensional deep detection items, including hardware basic computing power detection, GPU memory subsystem detection, kernel scheduling performance detection, inter-node full-topology communication detection, system environment and driver configuration consistency verification, training scenario simulation detection, etc. In alternative implementations, detection items and execution order relationships can also be generated using preset rules, heuristic rule engines, genetic algorithms, PPO deep reinforcement learning models, or multi-objective optimization algorithms.

[0104] During the detection execution phase, the first and second detection items are determined from multiple detection items based on their execution order. Specifically, this is achieved by constructing a detection execution Directed Acyclic Graph (DAG): each detection item is treated as a node in the DAG; the pre-dependencies between detection items are analyzed, and directed edges are added between nodes with dependencies. Detection items without pre-dependencies (i.e., zero in-degree in the DAG) are identified as the first detection item, and detection items with pre-dependencies (i.e., in-edges in the DAG) are identified as the second detection item. In alternative implementations, a dependency matrix or task flow topology table can also be used to identify dependency-free detection items.

[0105] Subsequently, each first detection item is executed in parallel, and each second detection item is executed sequentially according to its pre-dependency. Specifically, a multi-threaded scheduler (e.g., 4-thread, 8-thread, or 16-thread) is started, simultaneously executing all first detection items without pre-dependencies; for second detection items with pre-dependencies, subsequent detection items are started sequentially after the pre-dependency item has been executed and passed. During parallel scheduling, if a pre-dependency item fails, all subsequent related detection items in that link are immediately terminated to avoid unnecessary time consumption. Taking a 32-node large model training task as an example, the simultaneous parallel execution of four first detection items—basic computing power detection, GPU memory subsystem detection, kernel scheduling detection, and system configuration verification—takes approximately 2 minutes; then, after the system configuration verification passes, full-topology communication detection is started, taking approximately 8 minutes; finally, after all hardware detection items pass, scene simulation detection is started, taking approximately 1 minute, for a total full detection time of approximately 11 minutes. Taking a single-node CNN small model training task as an example, the simultaneous parallel execution of all first detection items takes approximately 22 seconds.

[0106] An adaptive timeout control mechanism is also implemented during the execution of detection items. Specifically, timeout thresholds for each detection item are dynamically allocated based on the user-defined maximum detection duration: larger timeout thresholds are assigned to detection items expected to take longer, and smaller timeout thresholds are assigned to those expected to take shorter. When the execution duration of any detection item exceeds the corresponding timeout threshold, the execution of that detection item is terminated and a warning is generated, while other detection items continue to be executed. Simultaneously, the total execution duration is monitored in real time. When the difference between the maximum detection duration and the total execution duration is less than a preset duration threshold (e.g., less than 1 minute remaining), a dynamic pruning mechanism is triggered. Detection items with a priority lower than the preset priority threshold are removed from the remaining detection items (e.g., removing the full simulation detection of the training scene with priority P2), and only core detection items continue to be executed, ensuring that core detections are completed within the maximum detection duration.

[0107] During the execution of the detection items, a rapid termination and fault isolation mechanism is also included. Specifically, the detection results of each detection item are collected in real time. When any detection result indicates the existence of a preset fault (i.e., an unrecoverable severe fault, including hardware failures such as GPU card drop, uncorrectable memory errors, configuration inconsistencies such as GPU driver version and CUDA version mismatch, container image loading failure, communication link interruption such as RDMA connection disconnection between nodes, network card failure, etc.), all unexecuted detection items are immediately terminated and a detection failure result is output. When the detection result indicates the existence of a faulty node but the number of faulty nodes does not exceed a preset fault tolerance threshold, the execution of the detection items that need to be executed on the faulty node is canceled, and other detection items continue to be executed; when the number of faulty nodes exceeds the preset fault tolerance threshold, all unexecuted detection items are terminated and a detection failure result is output.

[0108] Before executing the detection items, an incremental detection mechanism is also in place. On one hand, a state fingerprint reuse mechanism can be used: query the first historical detection results of the training task. When the training task and the historical training task meet the reuse conditions (i.e., the physical node set is the same and the network topology has not changed), the state fingerprint verification determines the state corresponding to each detection item. State fingerprint verification includes: collecting the unique hardware identifier of the node to generate a hardware hash value (including GPU UUID, card type, memory capacity, node SN code, network card MAC address, CPU ID, etc.), collecting the software configuration to generate an environment hash value (including GPU driver version, CUDA version, cuDNN version, system kernel version, container image ID, deep learning framework version, key environment variables, etc.), and comparing the current hash value with the historical hash value. For detection items whose state has not changed, the historical detection results are directly reused; for detection items whose state has changed, the detection is re-executed. On the other hand, a validity-based hierarchical reuse mechanism can be adopted: The second historical detection results of the training task are queried (including sub-detection results of each detection item and their corresponding storage duration). When the reuse conditions are met, the validity threshold corresponding to each detection item is obtained (e.g., 24 hours for hardware basic detection, 2 hours for environment configuration detection, and 30 minutes for communication detection). The storage duration is compared with the validity threshold. For detection items whose storage duration does not exceed the validity threshold, the historical results are directly reused. For detection items whose storage duration exceeds the validity threshold, the detection is re-executed. The above two incremental detection mechanisms can be used in combination. For example, the status fingerprint can be used to determine whether there has been a change. If there has been no change, the storage duration can be further determined to be within the validity period.

[0109] After all detection items are completed, the detection operation data for this detection process is collected (including the actual time consumed by each detection item, fault detection status, detection coverage, etc.). After the training task starts, the operational stability data of the training task is collected during the operation: an observation window is set according to the expected operation cycle of the task (the first 30 minutes after training starts for short-cycle tasks, and the first hour after training starts for long-cycle tasks). Within this window, hardware fault indicators (such as the number of GPU XID errors, the number of GPU frequency reductions, and the number of memory errors), communication fault indicators (such as the number of inter-node communication timeouts, the number of 3D parallel synchronization failures, and the number of link disconnections), and training convergence indicators (such as whether the loss diverges and the training throughput jitter rate) are collected. Then, the reinforcement learning model is retrained using the detection operation data and the operational stability data. Specifically, a composite update mechanism is adopted, which combines basic immediate reward with delayed stability correction reward: after detection is completed, the basic reward is calculated based on the detection coverage and detection time; after the observation window ends, the correction reward ΔR is calculated based on the operational stability data; the network parameters of the DQN model are updated with total reward = basic reward + ΔR, so that the model can pursue high efficiency in the detection stage while taking into account the long-term stability in the training stage.

[0110] Finally, the execution results of all detection items are summarized to complete the global fault determination and output the final detection result. If the detection passes, the training task is allowed to start; if the detection fails, fault details and location information are output, and all data from this detection are collected for model iteration and optimization.

[0111] In summary, the task detection method provided in this application, through mechanisms such as adaptive planning of the detection process, resolution and parallel scheduling of detection item dependencies, adaptive timeout control, rapid termination and fault isolation, incremental detection reuse, and model iterative optimization, can achieve differentiated scheduling based on the pre-dependencies between detection items. This allows detection items without pre-dependencies to be executed in parallel, avoiding the linear increase in time consumption caused by the forced serial execution regardless of the existence of dependencies in the prior art. Thus, while ensuring the correctness of the detection logic, the overall detection time is significantly shortened, and the detection strategy is self-learned and self-optimized.

[0112] Figure 4 This is a block diagram illustrating an embodiment of a task detection device provided in this application. Figure 4 As shown, the device includes: Module 41 is used to acquire the task features and constraints of the training task; The generation module 42 is used to generate multiple detection items and the execution order relationship between the detection items according to the task characteristics and the constraints. The determining module 43 is used to determine a first detection item and a second detection item from a plurality of detection items according to the execution order relationship between the detection items, wherein the first detection item is a detection item without a prerequisite dependency determined according to the execution order relationship, and the second detection item is a detection item with a prerequisite dependency determined according to the execution order relationship. The execution module 44 is used to execute each of the first detection items in parallel, and to execute each of the second detection items sequentially according to the prerequisite dependencies of each of the second detection items.

[0113] In one possible implementation, the generation module is specifically used for: The task features and constraints are input into a pre-trained reinforcement learning model, wherein the reinforcement learning model is pre-trained with the longest detection time and fault tolerance as constraints, and the optimization objectives being to maximize detection coverage and minimize detection time. Obtain the multiple detection items output by the reinforcement learning model and the execution order relationship between the detection items.

[0114] In one possible implementation, the device further includes a training module for: After all test items have been completed, collect the test operation data for this test process; After starting the training task, collect the operational stability data of the training task during its execution. The reinforcement learning model is retrained using the operational stability data and the detection operational data.

[0115] In one possible implementation, the device further includes a stop module for: During the execution of each of the aforementioned detection items, real-time detection results for each of the aforementioned detection items are collected; When any real-time detection result indicates the existence of a preset fault, all unexecuted detection items are terminated, and a detection failure result is output. The preset fault is an unrecoverable fault. When the real-time detection result indicates the existence of a faulty node and the number of faulty nodes does not exceed the preset fault tolerance threshold, the execution of the detection item that needs to be executed on the faulty node is cancelled, and other detection items are executed. When the number of faulty nodes exceeds the preset fault tolerance threshold, all unexecuted detection items are terminated, and a detection failure result is output.

[0116] In one possible implementation, the device further includes a first multiplexing module for: Before executing each detection item, the first historical detection result of the training task is queried, and the first historical detection result includes the first sub-detection result of each detection item. When the training task and the historical training task corresponding to the first historical detection result meet the reuse condition, the state corresponding to each detection item is determined. The state is used to characterize whether the node hardware configuration and software environment on which the detection item depends have changed. For the third detection item whose state has not changed, query the first sub-detection result corresponding to the third detection item from the first historical detection result, and use the first sub-detection result as the detection result of the third detection item in this detection. For the fourth detection item whose state has changed, execute the fourth detection item to obtain the detection result of the fourth detection item in this detection.

[0117] In one possible implementation, the device further includes a second multiplexing module for: Before executing each detection item, the second historical detection result of the training task is queried. The second historical detection result includes the second sub-detection result of each detection item and the corresponding storage time. When the training task and the historical training task corresponding to the second historical detection result meet the reuse condition, obtain the validity period threshold corresponding to each detection item. The storage duration of the second sub-detection result of each detection item is compared with the corresponding validity period threshold; For the fifth detection item whose storage duration has not exceeded the validity period threshold, the second sub-detection result of the fifth detection item is reused as the detection result of this detection. For the sixth detection item where the storage duration exceeds the validity period threshold, the sixth detection item is executed.

[0118] In one possible implementation, the device further includes a removal module for: During the execution of each of the aforementioned detection items, a timeout threshold corresponding to each of the aforementioned detection items is obtained, wherein the timeout threshold is dynamically allocated based on the longest detection duration; When the execution time of any detection item exceeds the corresponding timeout threshold, the execution of the detection item is terminated and a warning sign is generated, and the execution of other detection items continues; When the difference between the longest detection duration in the constraints and the total detection duration already executed is less than a preset duration threshold, detection items with a priority lower than the preset priority threshold are removed from the remaining detection items.

[0119] like Figure 5As shown in the figure, this application provides a device including a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114. Memory 113 is used to store computer programs; In one embodiment of this application, when the processor 111 executes a program stored in the memory 113, it implements the task detection method provided in any of the foregoing method embodiments, including: Obtain the task characteristics and constraints of the training task; Based on the task characteristics and the constraints, multiple detection items and the execution order relationship between the detection items are generated. Based on the execution order relationship between the various detection items, a first detection item and a second detection item are determined from the plurality of detection items, wherein the first detection item is a detection item without a prerequisite dependency determined according to the execution order relationship, and the second detection item is a detection item with a prerequisite dependency determined according to the execution order relationship. Each of the first detection items is executed in parallel, and each of the second detection items is executed sequentially according to their prerequisite dependencies.

[0120] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the task detection method provided in any of the foregoing method embodiments.

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0123] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0124] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A task detection method, characterized in that, The method includes: Obtain the task characteristics and constraints of the training task; Based on the task characteristics and the constraints, multiple detection items and the execution order relationship between the detection items are generated. Based on the execution order relationship between the various detection items, a first detection item and a second detection item are determined from the plurality of detection items, wherein the first detection item is a detection item without a prerequisite dependency determined according to the execution order relationship, and the second detection item is a detection item with a prerequisite dependency determined according to the execution order relationship. Each of the first detection items is executed in parallel, and each of the second detection items is executed sequentially according to their prerequisite dependencies.

2. The method according to claim 1, characterized in that, The step of generating multiple detection items and the execution order relationship between the detection items based on the task characteristics and the constraints includes: The task features and constraints are input into a pre-trained reinforcement learning model, wherein the reinforcement learning model is pre-trained with the longest detection time and fault tolerance as constraints, and the optimization objectives being to maximize detection coverage and minimize detection time. Obtain the multiple detection items output by the reinforcement learning model and the execution order relationship between the detection items.

3. The method according to claim 2, characterized in that, The method further includes: After all test items have been completed, collect the test operation data for this test process; After starting the training task, collect the operational stability data of the training task during its execution. The reinforcement learning model is retrained using the operational stability data and the detection operational data.

4. The method according to claim 1, characterized in that, The method further includes: During the execution of each of the aforementioned detection items, real-time detection results for each of the aforementioned detection items are collected; When any real-time detection result indicates the existence of a preset fault, all unexecuted detection items are terminated, and a detection failure result is output. The preset fault is an unrecoverable fault. When the real-time detection result indicates the existence of a faulty node and the number of faulty nodes does not exceed the preset fault tolerance threshold, the execution of the detection item that needs to be executed on the faulty node is cancelled, and other detection items are executed. When the number of faulty nodes exceeds the preset fault tolerance threshold, all unexecuted detection items are terminated, and a detection failure result is output.

5. The method according to claim 1, characterized in that, The method further includes: Before executing each detection item, the first historical detection result of the training task is queried, and the first historical detection result includes the first sub-detection result of each detection item. When the training task and the historical training task corresponding to the first historical detection result meet the reuse condition, the state corresponding to each detection item is determined. The state is used to characterize whether the node hardware configuration and software environment on which the detection item depends have changed. For the third detection item whose state has not changed, query the first sub-detection result corresponding to the third detection item from the first historical detection result, and use the first sub-detection result as the detection result of the third detection item in this detection. For the fourth detection item whose state has changed, execute the fourth detection item to obtain the detection result of the fourth detection item in this detection.

6. The method according to claim 1, characterized in that, The method further includes: Before executing each detection item, the second historical detection result of the training task is queried. The second historical detection result includes the second sub-detection result of each detection item and the corresponding storage time. When the training task and the historical training task corresponding to the second historical detection result meet the reuse condition, obtain the validity period threshold corresponding to each detection item. The storage duration of the second sub-detection result of each detection item is compared with the corresponding validity period threshold; For the fifth detection item whose storage duration has not exceeded the validity period threshold, the second sub-detection result of the fifth detection item is reused as the detection result of this detection. For the sixth detection item where the storage duration exceeds the validity period threshold, the sixth detection item is executed.

7. The method according to claim 1, characterized in that, The method further includes: During the execution of each of the aforementioned detection items, a timeout threshold corresponding to each of the aforementioned detection items is obtained, wherein the timeout threshold is dynamically allocated based on the longest detection duration; When the execution time of any detection item exceeds the corresponding timeout threshold, the execution of the detection item is terminated and a warning sign is generated, and the execution of other detection items continues; When the difference between the longest detection duration in the constraints and the total detection duration already executed is less than a preset duration threshold, detection items with a priority lower than the preset priority threshold are removed from the remaining detection items.

8. A task detection device, characterized in that, The device includes: The acquisition module is used to acquire the task features and constraints of the training task; The generation module is used to generate multiple detection items and the execution order relationship between the detection items based on the task characteristics and the constraints. The determining module is configured to determine a first detection item and a second detection item from a plurality of detection items based on the execution order relationship between the detection items, wherein the first detection item is a detection item without a prerequisite dependency determined based on the execution order relationship, and the second detection item is a detection item with a prerequisite dependency determined based on the execution order relationship. An execution module is configured to execute each of the first detection items in parallel, and to execute each of the second detection items sequentially according to their prerequisite dependencies.

9. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a task detection program stored in the memory to implement the task detection method according to any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the task detection method according to any one of claims 1-7.

Images