Training method and device of pre-training language model, computer equipment and medium

Abstract

The invention provides a training method and device of a pre-training language model, computer equipment and a medium. The method comprises the following steps: obtaining a training data combination for expressing the same semantics by using different languages; inputting the training data combination into a pre-training language model with different language understanding capabilities to enable the pre-training language model to pre-train the training data combination to obtain an output data combination corresponding to the training data combination, and calculating a loss value of the training data combination according to the output data combination; and updating model parameters of the pre-training language model by using the loss value so as to improve the similarity of the output data combination. According to the method, the downstream task processing performance of the natural language processing model in a low-resource language scene is improved, and the type range of downstream tasks applicable to the pre-training language model in a multi-language scene is widened.

Description

technical field [0001] The present disclosure relates to the field of computer technology, and more specifically, to a training method, device, computer equipment and media for a pre-trained language model. Background technique [0002] With the continuous development of deep learning technology and global internationalization, the trend of business internationalization is more obvious, and the demand for artificial intelligence (AI) technology that can handle multiple languages ​​is increasing day by day. Currently, in the field of natural language processing, pre-training language models is a way to achieve AI. For a natural language processing model suitable for specific downstream tasks covering different languages, the pre-trained language model needs to be fully fine-tuned with training sample data covering different languages ​​related to large-scale downstream tasks, while international business (for example, cross-border e-commerce business, cross-border logistics ...

AI Technical Summary

Problems solved by technology

For a natural language processing model suitable for specific downstream tasks covering different languages, the pre-trained language model needs to be fully fine-tuned with training sample data covering different languages ​​related to large-scale downstream tasks, while international business (for example, cross-border e-commerce business, cross-border logistics business and international business intelligent customer service, etc.) often cover different languages, and the sample data of low-resource languages ​​(minor languages) related to international business is usually relatively scarce, making the training sample data of low-resource languages ​​related to downstream tasks difficult The fine-tuning of the pre-trained language model is insufficient, which reduces the performance of the natural language processing model in low-resource language scenarios, and also makes it unsuitable for downstream tasks where low-resource language sample data is scarce by fine-tuning the pre-trained language model. Natural language processing model for downstream tasks, which reduces the range of downstream tasks that pre-trained language models can apply to in different language scenarios

Images