Unlock instant, AI-driven research and patent intelligence for your innovation.

Language model training method and device, electronic equipment and storage medium

A training method and language model technology, applied in natural language data processing, electrical digital data processing, special data processing applications, etc., can solve the problems of manual labeling, high training costs, low accuracy of language models, etc., to reduce training time The effect of time

Pending Publication Date: 2022-02-11
阳光保险集团股份有限公司
View PDF0 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, this training method will lead to a gap between the pre-training stage model and the downstream task fine-tuning stage model, making the model accuracy of the final language model lower, and this training method also requires a lot of manual labeling, which makes the training cost higher.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Language model training method and device, electronic equipment and storage medium
  • Language model training method and device, electronic equipment and storage medium
  • Language model training method and device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only It is a part of the embodiments of this application, not all of them. The components of the embodiments of the application generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments of the present application, every other embodiment obtained by those skilled in the art withou...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a language model training method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining initial text data related to a preset natural language processing task according to the preset natural language processing task; obtaining a text loading template which corresponds to the preset natural language processing task and is used for loading a training text sample; loading initial text data according to the text loading template to obtain a training text sample for training a language model, wherein the training text sample comprises a training text with identification information and a training text without identification information; and using the training text sample to iteratively update the initial language model to generate a target language model. According to the method, the training of the language model is synchronously carried out by using the identification information training text and the identification-free information training text, so that the model precision can be improved.

Description

technical field [0001] The present application relates to the technical field of natural language processing, in particular to a language model training method, device, electronic equipment and storage medium. Background technique [0002] The pre-trained language model is the basic research work of Natural Language Processing (NLP), which is widely used in various task scenarios such as text classification, semantic similarity, and entity recognition. After Google released the open source pre-trained language model BERT (Bidirectional Encoder Representations from Transformers) in 2019, research and applications in this field have become more and more popular. The standard paradigm currently used in models applied to various natural language tasks is pretraining + fine-tuning (Pretrain+Finetune), that is, first pretraining a language model on a large amount of unlabeled corpus, and then adding some modules such as fully connected layers to the model, Finetune (fine-tuning) ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/35G06F40/186G06F40/279
CPCG06F16/35G06F40/279G06F40/186
Inventor 张晗杜新凯吕超谷姗姗孙垚锋李文灏
Owner 阳光保险集团股份有限公司