Model training method and device, equipment, storage medium and program product

By using scaling parameters to convert input tensors and model parameters in the deep learning model training process, the problem of high computing resources consumption in deep learning model training is solved, and efficient model training and storage resource saving is achieved.

CN120509453AActive Publication Date: 2025-08-19MOORE THREADS TECH CO LTD
6 Cites 2 Cited by

Patent Information

Application Number
CN202511006917.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-07-21
Publication Date
2025-08-19
Estimated Expiration
2045-07-21

AI Technical Summary

Technical Problem

Currently, deep learning models consume high computing resources during training. Especially when using FP32 and FP16 data types, it is difficult to effectively use limited computing resources for large-scale model training.

Method used

By using scaling parameters to convert the input tensor and model parameters during model training, the low-precision first data type (such as FP8) is converted into a high-precision second data type (such as FP16 or FP32), and the corresponding type conversion is performed in the reverse process to unify the data type and use low-precision gradients to update the model parameters.

Benefits of technology

While ensuring the accuracy of the model, the model training efficiency is improved, the demand for computing power and storage resources is reduced, and more efficient model training is achieved.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a model training method and device, equipment, a storage medium and a program product, and the method comprises the steps: carrying out the type conversion of an input tensor of a second data type based on a zooming parameter corresponding to a linear layer in a forward process, and obtaining an input tensor of a first data type; after the calculation of the linear layer is completed, converting the output tensor of the first data type into an output tensor of a second data type; in a reverse process, performing type conversion on the model parameter of the first data type based on a scaling parameter corresponding to a linear layer to obtain a model parameter of a second data type; determining a gradient of the second data type based on the model parameter of the second data type and the loss value of the second data type; and updating model parameters of the model by using the gradient of the second data type to obtain a trained model. Therefore, under the condition that the model precision is guaranteed, the model training efficiency is improved, and the requirements for computing power and storage during model training are reduced.
Need to check novelty before this filing date? Find Prior Art