Language model training method, electronic device, storage medium and product

By performing pre-defined paradigm training and pseudo-label deterministic estimation on the pre-trained language model, the problem of low accuracy of the language model when there are few labeled labels is solved, and a high-precision language model can be trained with a small number of labeled labels.

CN116401364BActive Publication Date: 2026-06-16ALIBABA (CHINA) CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
ALIBABA (CHINA) CO LTD
Filing Date
2023-03-31
Publication Date
2026-06-16

AI Technical Summary

Technical Problem

Existing technologies result in low accuracy of language models when the number of labeled training samples is small, making it difficult to train high-precision language models.

Method used

The pre-trained language model is trained using a pre-defined paradigm with multiple first training sample corpora based on the target training task to obtain a teacher language model. The teacher language model is then used to identify multiple unlabeled second training sample corpora, calculate the deterministic values ​​of pseudo-labels, and select easily separable training sample corpora that meet the threshold conditions for training until a student language model with high accuracy is obtained.

Benefits of technology

With a small number of labeled tags, the accuracy of the language model was improved. Through multiple rounds of training and deterministic estimation of pseudo-labels, a high-precision language model was obtained.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116401364B_ABST
    Figure CN116401364B_ABST
Patent Text Reader

Abstract

The application provides a language model training method, an electronic device, a storage medium and a product, and belongs to the technical field of artificial intelligence. The method comprises the following steps: performing preset paradigm training on a pre-training language model based on a first training sample corpus to obtain a teacher language model; calling the teacher language model to perform identification on a second training sample corpus to obtain a category probability distribution of the second training sample corpus; calculating a certainty value of a pseudo label of the second training sample corpus based on the category probability distribution of the second training sample corpus; obtaining easy-to-classify training sample corpora whose certainty values meet threshold conditions from the second training sample corpus based on the certainty value of the pseudo label of the second training sample corpus; and performing preset paradigm training on the teacher language model based on the easy-to-classify training sample corpora to obtain a student language model that completes a target training task. The application can train a language model with high model precision based on a small amount of training sample corpora with labeled labels.
Need to check novelty before this filing date? Find Prior Art