Model fine tuning method, device, equipment, medium and product
By generating the input vocabulary list and output vocabulary list of some tokens, and only updating model parameters that match the fine-tuning requirements, the problems of large memory usage and slow training speed in the existing technology are solved, and the memory usage and training speed are reduced.
CN119692496APending Publication Date: 2025-03-25SHANGHAI KAIYONG INFORMATION TECHNOLOGY CO LTD
0 Cites 0 Cited by
Patent Information
- Application Number
- CN202411739052.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-28
- Publication Date
- 2025-03-25
AI Technical Summary
Technical Problem
In the fine-tuning process of pre-training language models, the prior art cannot effectively reduce the memory usage and improve the training speed, especially when fine-tuning the input layer or output layer parameters are required.
Method used
By generating an input vocabulary and output vocabulary containing some tokens, only model parameters that match the fine-tuning requirements are updated, thereby reducing memory usage and improving training speed.
Benefits of technology
It effectively reduces the video memory usage in the model fine-tuning stage, improves the model training speed, and avoids the problem of decreased training speed and batch processing capability caused by insufficient video memory.
✦ Generated by Eureka AI based on patent content.
Abstract
The invention provides a model fine tuning method and device, equipment, a medium and a product, and the method comprises the steps: obtaining training data, and obtaining a first model fine tuning demand; generating a first input word list according to the first character set and the second character set; for each training step, executing the following operations: inputting a first character sequence corresponding to the current training step into a pre-training language model, and receiving an output result of the pre-training language model; and in response to the first target character hitting the first sub-word list, updating model parameters of the pre-training language model according to the output result. The method can reduce the video memory occupancy in the model fine tuning stage and improve the model training speed.
Need to check novelty before this filing date? Find Prior Art