Method, device, device and computer storage medium for training semantic representation model

Abstract

The application discloses a method, device, device and computer storage device for training a semantic representation model, and relates to the technical field of natural language processing in artificial intelligence. The specific implementation plan is: obtain the semantic representation model trained for the first language as the first semantic representation model; use the bottom layer and top layer of the first semantic representation model as the trained layer, initialize the trained layer, and keep other The model parameters of the layer remain unchanged, and the trained layer is trained with the training corpus of the second language until the training end condition is reached; the untrained layers are added to the trained layer one by one from bottom to top, and are respectively executed: keep dividing the trained layer The model parameters of other layers other than the layer remain unchanged, and the training layer is trained with the training corpus of the second language until the training end conditions are reached respectively; after all the layers are trained, the semantic representation model for the second language is obtained. The application can reduce the cost, and the training efficiency is higher.

Description

technical field [0001] This application relates to the field of computer application technology, in particular to artificial intelligence technology. Background technique [0002] This year, the pre-training model represented by the BERT (Bidirectional Encoder Representation from Transformers) model has greatly improved the effect of NLP (Natural Language Processing, Natural Language Processing) tasks. However, the current mainstream semantic representation models focus on common languages ​​such as English, Chinese, French, and German. However, there are as many as thousands of languages ​​in the world, most of which have less corpus than English and other common languages. We call them low-resource languages. The training of pre-trained models requires a large amount of computing resources and is expensive. The cost of each model is as high as hundreds of thousands or even millions. Therefore, it is difficult to construct enough corpus for each language for training. And...

AI Technical Summary

Problems solved by technology

The training of the pre-training model requires a lot of computing resources and is expensive. The cost of each model is as high as hundreds of thousands or even millions. Therefore, it is difficult to build enough corpus for each language for training

And for languages ​​with very scarce corpus such as Czech, it is even difficult to collect enough corpus for training

Images