Abnormal convergence detection method based on distributed training communication feature distribution drift

By constructing a joint distribution model of distributed training communication features and using distribution drift analysis to identify potential anomalies, the problem of difficulty in early detection of anomaly convergence in existing technologies is solved, and effective early warning and fault location are achieved in distributed training.

CN122133760APending Publication Date: 2026-06-02BEIJING WANGSHEN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
BEIJING WANGSHEN TECH CO LTD
Filing Date
2026-03-03
Publication Date
2026-06-02

AI Technical Summary

Technical Problem

Existing technologies struggle to detect abnormal convergence in advance during distributed deep learning training, especially when minor anomalies in the communication process cause overall training blockage or failure. Furthermore, existing methods primarily rely on single performance metrics or communication latency monitoring, lacking effective early warning mechanisms.

Method used

By collecting communication data between distributed training nodes, multidimensional features are extracted and a joint feature distribution model is constructed. Distribution drift analysis is used to identify potential anomalies during training, including data packet length, timestamps, and quintuple information. Combined with metrics such as KL divergence and JS divergence, early warning and location of anomaly convergence are achieved.

Benefits of technology

Without intruding on the training framework, it can achieve early warning and location analysis of abnormal convergence through passive communication traffic monitoring, supports fault diagnosis, and is suitable for various distributed training scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122133760A_ABST
    Figure CN122133760A_ABST
Patent Text Reader

Abstract

The application discloses a kind of based on distributed training communication feature distribution drift's abnormal convergence detection method, belong to high-performance computing and artificial intelligence training process monitoring technical field.The method of the present application first extracts the data packet between distributed training nodes, calculates the feature of data packet to form multidimensional feature set and constructs feature joint distribution model, calculates the baseline distribution of communication behavior based on joint distribution model in training normal operation stage, then constructs current distribution based on joint distribution model in training process, using distribution difference degree measurement method calculates the drift degree of current distribution relative to baseline distribution, when drift degree is greater than preset threshold, it is judged that the training job exists abnormal convergence risk, and the abnormal communication interval is positioned.The method of the present application can be monitored relying on passive communication flow, without invading training framework, method can be early warning before training failure or convergence anomaly occurs, applicable to a variety of distributed training and high-performance computing scenarios.
Need to check novelty before this filing date? Find Prior Art