A large model association fault positioning method and system for heterogeneous training scenarios

Abstract

The application relates to a large model connection fault positioning method and system for a heterogeneous training scene, and the method comprises the following steps: periodically polling a cluster management platform to monitor and identify a new training task and metadata thereof, and parallelly capturing a connection data packet thereof and extracting connection key parameters, obtaining an original data stream after association, generating a structured feature sequence by performing real-time processing and feature extraction on the original data stream; comparing the structured feature sequence with an expected norm to determine and locate a fault node, classifying and differentially analyzing the fault node according to its propagation characteristics to obtain analysis strategy decisions of each connection fault; constructing a fault chain based on a specific node by adopting a hierarchical aggregation strategy, and performing causal analysis on the fault chain by a structure causal model according to the analysis strategy decisions of each connection fault to obtain fault roots of each connection fault. Compared with the prior art, the application realizes non-invasiveness of fault positioning, improves the accuracy of fault positioning and reduces the operation overhead.

Images