A Chinese-oriented pre-training method and system

Abstract

The invention relates to a Chinese-oriented pre-training method and system, comprising: a model parameter configuration module, a pre-training model generation module and a service packaging module. Model parameter configuration module: mainly aimed at the situation where users need to customize the Chinese pre-training model according to their own needs, so that they can configure the parameters of the pre-training model in a friendly interface; pre-training model generation module: configure and pre-train models according to the model parameters submitted by users Training corpus data, training a Chinese pre-training model, and saving it as a model file; service packaging module: packaging the model file into a Chinese feature extractor service, and providing users with corresponding Docker images to facilitate service deployment. The invention specifically aims at large-scale unsupervised Chinese corpus, and proposes a Chinese-oriented pre-training method and system, which effectively improves the performance of the pre-training method on Chinese tasks.

Description

technical field [0001] The invention relates to a Chinese-oriented pre-training method and system, belonging to the technical field of natural language processing. Background technique [0002] Generally speaking, most deep learning-based natural language processing tasks can be divided into the following three modules: data processing, text representation, and task-specific models. Among them, the data processing module and the specific task model module need to be designed according to the specific tasks, while the text representation module can be used as a relatively general module. Therefore, pre-training a general text representation module to reuse text features is of great significance for text transfer learning. In recent years, with the rapid development of deep learning methods, important breakthroughs have been made in many aspects such as machine translation, machine reading comprehension, and named entity recognition in the field of natural language processing...

AI Technical Summary

Problems solved by technology

[0005] (1) The large differences in Chinese and English language characteristics lead to unsatisfactory model effects

Most of the pre-training models are designed for English, and the language characteristics of English are more or less considered in terms of the network structure of the model, training methods, and downstream task application methods. Big difference, if it is directly transferred to the Chinese field, the effect is often not ideal

[0006] (2) The available Chinese unsupervised corpus has obvious classification phenomenon, and there are large differences in data size, quality, and field, and it is difficult to make full use of it

Generally speaking, relatively poor-quality corpora are large in scale, and high-quality corpora are mostly small in scale, which is not enough to support the training of huge pre-training models. If these corpora with large differences in quality are treated equally, they can be mixed for training. model, relatively poor-quality corpus may dilute the effect of high-quality corpus

Images