Hybrid expert model training system based on layer freezing

By using layer freezing technology and word grouping algorithm in the hybrid expert model training system, the layer freezing problem in the existing technology during the hybrid expert model training process is solved, and the model accuracy is maintained and the calculation cost is reduced.

CN119647560AActive Publication Date: 2025-03-18XIAMEN UNIV
5 Cites 3 Cited by

Patent Information

Application Number
CN202411692716.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-25
Publication Date
2025-03-18
Estimated Expiration
2044-11-25

AI Technical Summary

Technical Problem

The prior art is difficult to effectively freeze layer during the training of hybrid expert models, resulting in excessive model accuracy loss and computational cost, and is not suitable for hybrid expert models with dynamic loads.

Method used

A hybrid expert model training system based on layer freezing is proposed, using the expert's selection of path change rate as the model convergence index, freeze the network layer by layer, and calculate expert popularity through word grouping algorithm, optimize resource allocation and reduce communication overhead.

Benefits of technology

While maintaining the accuracy of the model, it reduces backward propagation computing and full-to-full communication, realizes equipment load balancing and efficient allocation of computing resources, and reduces training computing costs.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a hybrid expert model training system based on layer freezing, which relates to the field of model training and comprises an external memory, a freezing controller, a resource allocation controller and a plurality of computing devices. The external memory stores a data set for training the hybrid expert model; the freezing controller performs freezing decision making according to a path selected by an expert and sends the decision to the computing device for execution; the resource allocation controller executes a lexical grouping algorithm based on a frozen layer expert selection path to calculate expert popularity, and allocates computing resources according to the expert popularity; and the computing device stores the hybrid expert model, executes model training operation, executes a layer freezing task according to a layer freezing decision, and feeds back a freezing layer expert selection path and an expert selection path to the resource allocation controller and the freezing controller respectively. According to the method, training of the hybrid expert model is performed through a layer freezing technology, so that the training cost is reduced while the model precision is kept; computing resource allocation is carried out based on a word source grouping algorithm, and equipment load balancing is achieved.
Need to check novelty before this filing date? Find Prior Art