End-to-end language model pre-training method and system, equipment and storage medium

Abstract

The invention provides an end-to-end language model pre-training method and system, equipment and a storage medium. The method comprises the steps: according to a preset knowledge similarity judgment rule, retrieving existing knowledge fragments similar to input knowledge fragments in knowledge from an existing knowledge base; splicing the input knowledge segment and the retrieved existing knowledge segment to obtain a spliced knowledge segment; carrying out mask processing on the spliced knowledge fragments; and taking the masked spliced knowledge segments as input of language model pre-training to perform prediction training, and completing end-to-end language model pre-training. According to the method, the preset similarity judgment rule is utilized, retrieval of similar existing knowledge segments is carried out in the existing knowledge base through retrieval, the requirement of the model for parameters during training is reduced, and therefore the language model can enhance utilization of external knowledge based on retrieval, and the language model training efficiency is improved.

Description

technical field [0001] The present invention relates to a language model pre-training method, in particular to an end-to-end language model pre-training method, system, device and storage medium. Background technique [0002] Natural Language Processing (NLP), the purpose of which is to enable computers to have human listening, speaking, reading, writing, translation, questioning, answering, searching, summarizing, dialogue and chatting capabilities, and to use knowledge and common sense for reasoning and decision-making, and Support customer service, diagnosis, legal, teaching and other scenarios. Natural language understanding is considered the jewel in the crown of AI. Once there is a breakthrough, it will greatly promote the landing of AI in many important scenarios. [0003] The pre-training model is to make natural language processing enter the stage of large-scale and reproducible large-scale industrial application from the original stage of manual parameter adjustm...

AI Technical Summary

Problems solved by technology

[0008] Aiming at the huge demand for parameters and low training efficiency in the pre-training of the language model in the prior art, and at the same time increas

Images