Model prediction-based accelerated AI large model distributed training method and device

By introducing model prediction methods and convolution + channel attention mechanisms into the distributed training of large AI models, the problems of high communication overhead and accuracy loss in gradient compression technology are solved, achieving efficient model training and accuracy preservation.

CN117273120BActive Publication Date: 2026-04-17GUANGZHOU UNIVERSITY
0 Cites 0 Cited by

Patent Information

Application Number
CN202311119841.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-09-01
Publication Date
2026-04-17
Estimated Expiration
2043-09-01

AI Technical Summary

Technical Problem

Existing gradient compression techniques suffer from high communication overhead and loss of model accuracy in distributed training of large AI models, making it difficult to maintain model accuracy while reducing communication overhead.

Method used

We adopt a model-based prediction approach, utilizing a prediction model structure based on a PS parameter server and a convolutional + channel attention mechanism. By constructing a hierarchical parameter dataset, we train the prediction model and deploy it in the PS parameter server to predict the remaining layer parameters of the target model, thereby reducing communication volume and improving model prediction efficiency and robustness.

Benefits of technology

It significantly reduces communication volume, improves model accuracy and training efficiency, while maintaining model precision and generalization.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

This invention discloses a method and apparatus for accelerating distributed training of large AI models based on model prediction. The method includes: constructing a hierarchical parameter dataset; training a prediction model for predicting model parameters of a target model using an evaluation method based on prediction results; computing nodes updating the model parameters of some layers of the target model and then pushing them to a parameter server; on the parameter server, aggregating the received model parameters and predicting the model parameters of the remaining layers of the target model using the prediction model; concatenating the predicted and aggregated model parameters into a complete model parameter set and pushing it to all computing nodes; and replacing the local model parameters with the received complete model parameter set. The prediction model adopts a "convolution + channel attention mechanism + pooling" structure. This invention can significantly reduce communication volume without introducing a large amount of computational overhead while maintaining the accuracy of the target model.
Need to check novelty before this filing date? Find Prior Art