Model training method, text abstract generation method and related device

By introducing masked word prediction and sentence vector comparison tasks during pre-training, the language model is trained to understand the contextual relationships of dialogue text, which solves the problem that existing models cannot accurately represent text and improves the performance and training efficiency of downstream tasks.

CN116127316BActive Publication Date: 2026-07-03MASHANG CONSUMER FINANCE CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
MASHANG CONSUMER FINANCE CO LTD
Filing Date
2023-01-13
Publication Date
2026-07-03

AI Technical Summary

Technical Problem

Existing pre-trained language models are unable to understand and learn the contextual relationships in dialogues with fine granularity, resulting in poor performance on downstream tasks.

Method used

By using sample response texts and their associated context texts in the dialogue text as sample text pairs, a language model is trained. Masked word prediction, text prediction, and sentence vector comparison tasks of adjacent sentences are introduced to adjust the model parameters of the language model in order to improve the accuracy of text representation.

Benefits of technology

It improves the language model's ability to accurately represent text, thereby enhancing the performance of downstream tasks. Furthermore, by using a pre-trained language model, it accelerates the training process of the text summarization generation model, improving training efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116127316B_ABST
    Figure CN116127316B_ABST
Patent Text Reader

Abstract

The application discloses a model training method, a text abstract generation method and related equipment. The pre-training method comprises the following steps: obtaining a sample response text and context text associated with the sample response text in a dialogue text; encoding a masked text of the context text by using a language model to obtain a masked representation vector; predicting, based on the masked representation vector, prediction word information corresponding to a position of a masked word in the context text and a prediction response text associated with the context text; encoding a sentence in the context text by using the language model to obtain a sentence vector of the sentence, and determining a similarity between sentence vectors of adjacent sentences in the context text; and adjusting model parameters of the language model based on the masked word, the prediction word information, the prediction response text, the sample response text and the similarity between the sentence vectors of the adjacent sentences in the context text. Thus, the pre-trained language model obtained by training can accurately represent the text, which is beneficial to improving the effect of a downstream task.
Need to check novelty before this filing date? Find Prior Art