Training method, device, computer equipment and medium for pre-trained language model

Abstract

The present disclosure provides a training method, apparatus, computer equipment and medium for a pre-trained language model. The method includes: obtaining a training data combination that expresses the same semantics in different languages; inputting the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model obtains an output data combination corresponding to the training data combination after pre-training the training data combination, and calculates the loss value of the training data combination based on the output data combination; and using the loss value to update the model parameters of the pre-trained language model to improve the similarity of the output data combination. The present disclosure improves the performance of natural language processing models in processing downstream tasks in low-resource language scenarios and the range of types of downstream tasks that pre-trained language models can be applied to in multi-language scenarios.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a training method, apparatus, computer device, and medium for pre-training a language model. Background Art

[0002] With the continuous development of deep learning technology and global internationalization, the trend of business internationalization is more obvious, and the demand for artificial intelligence (AI) technology that can handle multiple languages ​​is increasing. At present, in the field of natural language processing, pre-trained language models are a method of implementing AI. For natural language processing models covering different languages ​​and suitable for specific downstream tasks, pre-trained language models need to be fully fine-tuned with training sample data covering different languages ​​related to large-scale downstream tasks, while international businesses (for example, cross-border e-commerce business, cross-border logistics business, and international business intelligent customer service, etc.) often cover different languages, and low-resource language (small language) sample data related to international business is usually scarce, which makes the training sample data of low-resource languages ​​related to downstream tasks insufficient for fine-tuning the pre-trained language model. This reduces the performance of the natural language processing model in processing downstream tasks in low-resource language scenarios, and also makes downstream tasks with scarce low-resource language sample data unsuitable for obtaining natural language processing models suitable for downstream tasks by fine-tuning the pre-trained language model, which reduces the range of downstream tasks that can be applied to the pre-trained language model in different language scenarios. Summary of the invention

[0003] In view of this, the present disclosure aims to improve the performance of natural language processing models in processing downstream tasks in low-resource language scenarios and the range of downstream tasks to which pre-trained language models can be applied in different language scenarios.

[0004] To achieve this objective, according to one aspect of the present disclosure, a method for training a pre-trained language model is provided, comprising:

[0005] Obtain a combination of training data that express the same semantics in different languages;

[0006] Inputting the training data combination into a pre-trained language model with different language understanding capabilities, so that after pre-training the training data combination, an output data combination corresponding to the training data combination is obtained, and a loss value of the training data combination is calculated according to the output data combination;

[0007] The loss value is used to update the model parameters of the pre-trained language model to improve the similarity of the output data combination.

[0008] Optionally, the training data combination includes at least a first text expressing a first semantic meaning in a first language and a second text expressing the first semantic meaning in a second language, the first text includes a plurality of first segmented words, and the second text includes a plurality of second segmented words.

[0009] Inputting the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model pre-trains the training data combination to obtain an output data combination corresponding to the training data combination, and calculating the loss value of the training data combination according to the output data combination includes:

[0010] Inputting the plurality of first word segmentations into the pre-trained language model, and obtaining a plurality of first word segmentation vectors from the pre-trained language model;

[0011] Inputting the plurality of second word segmentations into the pre-trained language model, and obtaining a plurality of second word segmentation vectors from the pre-trained language model;

[0012] Based on the multiple first word segmentation vectors and the multiple second word segmentation vectors, a similarity loss value between the vector representation of the first text and the vector representation of the second text is calculated.

[0013] Optionally, inputting the training data combination into a pre-trained language model with different language understanding capabilities so that the pre-trained language model pre-trains the training data combination to obtain an output data combination corresponding to the training data combination further includes:

[0014] Inputting the first text into the pre-trained language model, and obtaining a first text prediction value from the pre-trained language model;

[0015] Inputting the second text into the pre-trained language model, and obtaining a second text prediction value from the pre-trained language model;

[0016] Calculating the loss value of the training data combination according to the output data combination also includes:

[0017] Calculating a prediction loss value of the first text based on the first text prediction value and the label value of the first text;

[0018] Based on the second text prediction value and the label value of the second text, a prediction loss value of the second text is calculated.

[0019] Optionally, before updating the model parameters of the pre-trained language model using the loss value, the method further includes:

[0020] Calculate the linear weighted sum of the similarity loss value, the prediction loss value of the first text, and the prediction loss value of the second text as the loss value of the training data combination.

[0021] Optionally, the training data combination includes a plurality of bilingual data pairs. Updating the model parameters of the pre-trained language model using the loss value includes:

[0022] Using the mean of the loss values of the plurality of bilingual data pairs, update the model parameters of the pre-trained language model using a deep learning optimization algorithm.

[0023] Optionally, calculating the similarity loss value between the vector representation of the first text and the vector representation of the second text based on the plurality of first token vectors and the plurality of second token vectors includes:

[0024] Calculate the mean of the plurality of first token vectors as the text vector of the first text;

[0025] Calculate the mean of the plurality of second token vectors as the text vector of the second text;

[0026] Calculate the vector similarity between the text vector of the first text and the text vector of the second text, and calculate the similarity loss value based on the vector similarity.

[0027] Optionally, inputting the plurality of first tokens into the pre-trained language model to obtain a plurality of first token vectors by the pre-trained language model includes:

[0028] Input the plurality of first tokens into the pre-trained language model, and output the plurality of first token vectors from the i-th layer of the pre-trained language model;

[0029] Inputting the plurality of second tokens into the pre-trained language model to obtain a plurality of second token vectors by the pre-trained language model includes:

[0030] Input the plurality of second tokens into the pre-trained language model, and output the plurality of second token vectors from the i-th layer of the pre-trained language model, where i < N and N is the total number of layers of the pre-trained language model.

[0031] Optionally, obtaining a training data combination expressing the same semantics in different languages includes:

[0032] Obtain the translated text of the training sample;

[0033] Perform tokenization on the original text and the translated text of the training sample to obtain the training data combination expressing the same semantics in different languages.

[0034] According to one aspect of the present disclosure, a training device for a pre-trained language model is provided, comprising:

[0035] A training data combination acquisition unit, used to obtain a training data combination that expresses the same semantics in different languages;

[0036] a loss value calculation unit, configured to input the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model obtains an output data combination corresponding to the training data combination after pre-training the training data combination, and calculate the loss value of the training data combination according to the output data combination;

[0037] A model parameter updating unit is used to update the model parameters of the pre-trained language model using the loss value to improve the similarity of the output data combination.

[0038] According to one aspect of the present disclosure, there is provided a computer device, comprising:

[0039] A memory for storing computer executable code;

[0040] The processor is used to execute the computer executable code to implement the method as described above.

[0041] According to one aspect of the present disclosure, a computer-readable medium is provided, comprising a computer-executable code, wherein the computer-executable code implements the method described above when executed by a processor.

[0042] In the disclosed embodiment, the characteristic that the training data combination expressing the same semantics in different languages ​​is the corpus of the downstream task with the same semantics but expressed in different languages ​​is utilized. In the fine-tuning process of the pre-trained language model, the pre-trained language model obtains the output data combination corresponding to the training data combination after pre-training the training data combination, calculates the loss value of the training data combination using the output data combination, and updates the model parameters of the pre-trained language model based on the loss value. In this way, the similarity of the output data combination is increased. Through multiple iterations, the pre-trained language model can express the training data combination expressing the same semantics in different languages ​​to a similar semantic space during the fine-tuning process, thereby improving the cross-language representation ability of the model. In this way, for low-resource languages, there is no need for training samples of specific downstream tasks. It is only necessary to provide a large number of training samples of high-resource languages ​​for specific downstream tasks, and fine-tune the pre-trained language model to obtain a natural language processing model that supports low-resource languages ​​and is suitable for processing specific downstream tasks, thereby improving the performance of the natural language processing model in processing downstream tasks in low-resource language scenarios, and also improving the range of types of downstream tasks that the pre-trained language model can be applied to in different language scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects, features and advantages of the present disclosure will become more apparent through the description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0044] Figure 1 A system architecture diagram is shown for use in a training method for a pre-trained language model according to an embodiment of the present disclosure.

[0045] Figure 2 A flowchart of a method for training a pre-trained language model according to an embodiment of the present disclosure is shown.

[0046] Figure 3 The figure shows the internal structure of a training device for pre-training a language model according to an embodiment of the present disclosure.

[0047] Figure 4 The data structure of a bilingual data pair according to an embodiment of the present disclosure is shown.

[0048] Figure 5 A hardware diagram of a computer device that executes a training method for a pre-trained language model according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0049] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0050] Application scenarios and application architecture of the present disclosure

[0051] The application scenarios of the embodiments of the present disclosure may include forming a natural language processing model with different language understanding capabilities suitable for specific downstream tasks, so as to use the natural language processing model to complete natural language processing (NLP) in international business (for example, cross-border e-commerce business, cross-border logistics business, and international business intelligent customer service, etc.).

[0052] Figure 1 The system architecture diagram of the training method of the pre-trained language model according to the embodiment of the present disclosure is shown. The system architecture includes a training device 110, an execution device 120, a database 130 and a data storage system 140. The training device 110 includes a pre-trained language model 111 and a model fine-tuning device 112, and the execution device 120 includes a natural language processing model 121.

[0053] In the disclosed embodiment, the "pre-training-fine-tuning" mode is used to generate a natural language processing model with different language understanding capabilities suitable for specific downstream tasks. Natural language is human language, and natural language processing is the processing of human language. Natural language processing is a process of systematically analyzing, understanding and extracting information from text data in an intelligent and efficient way. Exemplarily, downstream tasks of natural language processing may include automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering system, and topic segmentation, etc. The pre-trained language model is a network model obtained by pre-training using a training data set. The pre-trained language model can be fine-tuned using a new training data set related to the downstream task according to different needs to obtain a natural language processing model suitable for different downstream tasks. Fine-tuning the pre-trained language model is to fine-tune the pre-trained language model using a new data set for a specific downstream task. Since the pre-trained language model needs to be fully pre-trained on a large-scale corpus, fine-tuning the pre-trained language model can achieve excellent performance on most natural language processing tasks.

[0054] In some embodiments, Figure 1As shown, the pre-trained language model 111 is a pre-trained model with different language comprehension. The pre-training method of the pre-trained language model 111 is as follows: a sample set consisting of a large number of training samples covering different languages ​​can be constructed, wherein each training sample includes a corpus represented by a language and a label pre-tagged for the corpus, and the label is the true value of the training sample. Each training sample in the training sample set is input into the pre-trained language model, and the pre-trained language model outputs the learned predicted value. The predicted value is compared with the label. If the output results of at least a predetermined proportion (for example, more than 95%) of the training samples in the training sample set are consistent with the label, it means that the model is pre-trained successfully. Otherwise, the parameters of each hidden node in each hidden layer in the model are adjusted so that the output results of at least a predetermined proportion (for example, more than 95%) of the training samples are consistent with the label. When the pre-trained language model is pre-trained successfully, it generally needs to undergo a testing process before it can be officially put into use. It can adopt the general testing method of the general machine learning model, so it is not repeated. At present, the common pre-trained models covering different languages ​​are XML, XML-R and VECO. The database 130 stores a set of training samples for fine-tuning the pre-trained language model 111. The database 130 can be embodied as a storage medium in any form. In the fine-tuning stage, the training device 110 calls the set of training samples stored in the database 130, and the model fine-tuning device 112 uses the set of training samples to iteratively train the pre-trained language model 111, and after the model converges, a natural language processing model 121 with different language comprehension is generated. The model fine-tuning device 112 is the main implementation component of the embodiment of the present disclosure. Since the process of the model fine-tuning device 112 fine-tuning the pre-trained language model 111 to generate the natural language processing model 121 will be described in detail below, it will not be repeated here.

[0055] In the inference stage, the execution device 120 can call the data, code, etc. related to the downstream task in the data storage system 140, or store the data, instructions, etc. in the data storage system 140. The data storage system 140 can be configured in the execution device 120, or it can be a memory outside the execution device 120. The execution device 120 calls the natural language processing model 121 to execute the downstream task. In some scenarios, the execution device 120 can be a translation device (for example, a smart terminal, a tablet computer, etc.). After the training device 110 generates a natural language processing model 121 suitable for a downstream translation task, in the inference stage, the execution device 120 can call the natural language processing model 121 to perform the translation task.

[0056] Training method of pre-trained language model according to an embodiment of the present disclosure

[0057] According to one embodiment of the present disclosure, a method for training a pre-trained language model is provided. The training method may be performed by a model fine-tuning device 112. In the case where the training device 110 is a single computer, the model fine-tuning device 112 is a part of the single computer, and the method for training the pre-trained language model is performed by a part of the single computer. In the case where the training device 110 is a set of multiple computers, the model fine-tuning device 112 may be a single computer therein, and the method for training the pre-trained language model is performed by the single computer. In the case where the training device 110 is in the form of a cloud, the model fine-tuning device 112 is a part of a series of computers or computers on the cloud, and the method for training the pre-trained language model is performed by a series of computers or parts of computers on the cloud.

[0058] like Figure 2 As shown, according to an embodiment of the present disclosure, a training method for a pre-trained language model includes: step S210, obtaining a training data combination that expresses the same semantics in different languages; step S220, inputting the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model obtains an output data combination corresponding to the training data combination after pre-training the training data combination, and calculates a loss value of the training data combination based on the output data combination; step S230, using the loss value to update the model parameters of the pre-trained language model to improve the similarity of the output data combination.

[0059] The above steps are described in detail below.

[0060] In step S210, a training data combination that uses different languages ​​to express the same semantics is obtained.

[0061] The downstream task is a specific task to be processed by the natural language processing model generated using the "pre-training-fine-tuning" mode, such as a sentence relationship extraction task, a translation task, a viewpoint role labeling task, an automatic viewpoint generation task, and a generative answer task. In order to generate a natural language processing model that supports more languages ​​and achieves good natural language processing effects on specific downstream tasks, it is necessary to use a training sample set of specific downstream tasks covering as many languages ​​as possible to fine-tune the pre-trained language model with different language understanding capabilities. The training sample set includes training samples covering different languages, wherein each training sample includes a corpus of a specific downstream task (e.g., a sentence relationship extraction task, a translation task, a viewpoint role labeling task, an automatic viewpoint generation task, and a generative answer task, etc.) expressed in a certain language, and a label pre-tagged for the corpus, which represents the true value of the training sample. The corpus in the available training sample set can be found or obtained from the Internet, and the specific expression of the corpus in the training sample set is not limited, for example, it can be expressed in the form of text, it can also be expressed in the form of voice, and it can also be expressed in the form of images. In one embodiment, the input of the pre-trained language model can only be training sentences expressed in text form. The training sentences expressed in speech form can be processed by speech recognition to be converted into training sentences expressed in text form. The training sentences expressed in image form can also be processed by image recognition to be converted into training sentences expressed in text form. Since the labeling technology is an existing technology, it will not be described in detail.

[0062] In some embodiments, the training sample includes a training data combination, where the training data combination expressing the same semantics in different languages ​​at least includes a first text expressing the first semantics in a first language and a second text expressing the first semantics in a second language, and the first language and the second language can be any two languages. The first text can include multiple first participles, and the second text can include multiple second participles. In some embodiments, the first text is the original text of the corpus of the training sample, and the second text can be the translated text of the corpus of the training sample. In some embodiments, the first text and the second text constitute a bilingual data pair. For example, the first text is "This trip needs careful planning", and the second text is "The trip needs careful planning", then "This trip needs careful planning" and "The trip needs careful planning" can be regarded as a bilingual data pair, which is a Chinese-English parallel language pair. It should be understood that the bilingual data pair here can also be a parallel language pair of low-resource languages ​​(Korean-Thai parallel language pair, Vietnamese-German parallel language pair, etc.).

[0063] In one embodiment, step S210 includes: obtaining a translation text of a training sample; performing word segmentation processing on the original text and the translation text of the training sample to obtain the training data combination that uses different languages ​​to express the same semantics.

[0064] The translation of the original text of the training sample corpus into the target text can be implemented by a translation machine implemented by software and / or hardware, such as Figure 3 The translation unit 311 of the training sample corpus is input into the translation unit 311 to obtain the translation text of the training sample corpus. It should be understood that the present disclosure should not be limited to this, and any method of obtaining the translation text of the training sample corpus is within the protection scope of the present disclosure. In one example, Figure 4 As shown in the figure, the original text of the training sample corpus is "How can you prove it, Can you tell me how to prove it", and the label of the original text is "implied relationship". The translated text of the training sample corpus is "How can you prove it, Can you tell me how to prove it", and the label of the translated text is also "implied relationship". The original text of the training sample corpus "How can you prove it, Can you tell me how to prove it" and the translated text "How can you prove it, Can you tell me how to prove it" constitute the training data combination, which is a Chinese-English parallel language pair.

[0065] In one embodiment, the original text and the translated text of the training sample corpus are then segmented, which can be implemented by a software and / or hardware-implemented segmentation machine, for example Figure 3 The word segmentation unit 312 can divide the input text into multiple minimum semantic units (tokens). The original text of the training sample corpus is input into the word segmentation unit 312 to obtain multiple word segments of the original text, and the translation text of the training sample corpus is input into the word segmentation unit 312 to obtain multiple word segments of the translation text. Word segmentation processing has mature technology, so it is not repeated here. In an example, Figure 4As shown in the figure, the original text of the training sample corpus "How can you prove it, Can you tell me how to proveit" can be segmented into multiple words: "How", "can", "you", "prove", "it", "Can", "you", "tell", "me", "how", "to", "prove", "it". The translated text of the training sample corpus can be segmented into multiple words: "you", "how", "can", "prove", "you", "can", "tell", "me", "how", "prove", "do".

[0066] During the fine-tuning process, the training data combination of each training sample in the training sample set is input into the pre-trained language model, and the pre-trained language model outputs the learned output data combination. The output data combination is compared with the corresponding label respectively. If the output results of at least a predetermined proportion (for example, more than 95%) of the training samples in the training sample set are consistent with the label, it means that the model is fine-tuned successfully. Otherwise, the training method of the pre-trained language model of the embodiment of the present disclosure is used to adjust the parameters of each hidden node in each hidden layer in the model, so that at least a predetermined proportion (for example, more than 95%) of the output results of the training samples are consistent with the label. When the pre-trained language model is fine-tuned successfully, it generally needs to undergo a testing process before it can be officially put into use. It can adopt the general testing method of the general machine learning model, so it is not repeated.

[0067] In step S220, the training data combination is input into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model obtains an output data combination corresponding to the training data combination after pre-training the training data combination, and calculates the loss value of the training data combination based on the output data combination.

[0068] The pre-trained language model here is pre-trained on a large-scale corpus covering different languages, and the pre-trained language model mainly learns the general basic knowledge of language processing. The above-mentioned training data combination is input into the pre-trained language model, and the pre-trained language model can obtain the output data combination corresponding to the training data combination after pre-training the training data combination. In one example, the original text "How can you prove it, Can you tell me how to prove it" of the training sample corpus is input into the pre-trained language model to obtain the predicted value of the original text (which can be one of the implications, contradictions or irrelevance). The translated text "How can you prove, can you tell me how to prove it" of the training sample corpus is input into the pre-trained language model to obtain the predicted value of the translated text (which can be one of the implications, contradictions or irrelevance). In one example, the word segmentation "How", "can", "you", "prove", "it", "Can", "you", "tell", "me", "how", "to", "prove", "it" of the original text are respectively input into the pre-trained language model, and the corresponding word segmentation vectors of the original text can be obtained by the pre-trained language model. The word segmentations "you", "how", "can", "prove", "you", "can", "tell", "me", "how", "prove", and "do" of the translation text are respectively input into the pre-trained language model, and the corresponding word segmentation vectors of the translation text can be obtained from the pre-trained language model.

[0069] In a deep neural network, the loss function reflects the performance of the algorithm by calculating the difference between the model prediction value and the true value. Therefore, a set of model parameters that minimize the loss function value can be found by training a pre-trained language model to obtain a natural language processing model with optimal performance. In some embodiments, the loss function value can be composed of the following parts: the prediction loss value of the original text of the corpus of the training sample, the prediction loss value of the translated text of the corpus of the training sample, and the similarity loss value of the training data combination. The prediction loss value of the original text of the corpus of the training sample is the loss value between the true label value of the original text and the model prediction value. The prediction loss value of the translated text of the corpus of the training sample is the loss value between the true label value of the translated text and the model prediction value. The similarity loss value of the training data combination is the loss value between the model prediction value of the original text and the model prediction value of the translated text.

[0070] In some embodiments, the prediction loss value of the original text of the corpus of the training sample can be implemented by a calculator implemented by software and / or hardware, for example Figure 3The first text prediction loss value calculation unit 321. In some embodiments, the calculation method of the prediction loss value of the original text of the corpus of the training sample includes: inputting the original text of the corpus of the training sample into a pre-trained language model, and obtaining the prediction value of the original text of the corpus of the training sample from the pre-trained language model; based on the prediction value of the original text of the corpus of the training sample and the label value of the original text of the corpus of the training sample, the prediction loss value of the original text of the corpus of the training sample is calculated. In one embodiment, a more common cross entropy loss function can be used as the prediction loss value of the original text of the corpus of the training sample. The calculation formula for the prediction loss value of the original text of the corpus of the training sample is as follows:

[0071]

[0072] in, is the prediction loss value of the original text of the training sample corpus, Y1 is the label value of the original text of the training sample corpus, is the predicted value of the original text of the training sample corpus, is the predicted value of the jth word segmentation of the original text of the training sample corpus.

[0073] It should be understood that the smaller the prediction loss value of the original text of the corpus of the training sample, the higher the prediction accuracy of the pre-trained language model for the original text.

[0074] In some embodiments, the prediction loss value of the translation text of the corpus of the training sample can be implemented by a calculator implemented by software and / or hardware, for example Figure 3 The second text prediction loss value calculation unit 322. In some embodiments, the calculation method of the prediction loss value of the translated text of the corpus of the training sample includes: inputting the translated text of the corpus of the training sample into a pre-trained language model, and obtaining the prediction value of the translated text of the corpus of the training sample from the pre-trained language model; based on the prediction value of the translated text of the corpus of the training sample and the label value of the translated text of the corpus of the training sample, calculating the prediction loss value of the translated text of the corpus of the training sample. In one embodiment, a more common cross entropy loss function can be used as the prediction loss value of the translated text of the corpus of the training sample. The calculation formula for the prediction loss value of the translated text of the corpus of the training sample is as follows:

[0075]

[0076] in, is the prediction loss value of the translation text of the training sample corpus, Y2 is the label value of the translation text of the training sample corpus, is the predicted value of the translation text of the training sample corpus, is the predicted value of the jth word segmentation of the translation text of the training sample corpus.

[0077] It should be understood that the smaller the prediction loss value of the translated text of the corpus of training samples, the higher the prediction accuracy of the pre-trained language model for the translated text.

[0078] In some embodiments, the similarity loss value of the training data combination can be implemented by a calculator implemented by software and / or hardware, such as Figure 3 the similarity loss value calculation unit 323. In some embodiments, the calculation method of the similarity loss value includes: inputting multiple word segments of the original text of the corpus of training samples into the pre-trained language model, and obtaining multiple word segment vectors corresponding to the original text from the pre-trained language model; inputting multiple word segments of the translated text of the corpus of training samples into the pre-trained language model, and obtaining multiple word segment vectors corresponding to the translated text from the pre-trained language model; calculating the similarity loss value between the vector representation of the original text of the corpus of training samples and the vector representation of the translated text of the corpus of training samples based on the multiple word segment vectors corresponding to the original text and the multiple word segment vectors corresponding to the translated text.

[0079] In some embodiments, the word segment vectors of the original text and the translated text of the corpus of training samples can be calculated in the intermediate layer of the pre-trained language model. The calculation methods of the word segment vectors of the original text and the translated text include: inputting multiple word segments of the original text of the corpus of training samples into the pre-trained language model respectively, and the i-th layer of the pre-trained language model outputs multiple first word segment vectors corresponding to the multiple word segments of the original text respectively; inputting multiple word segments of the translated text of the corpus of training samples into the pre-trained language model respectively, and the i-th layer of the pre-trained language model outputs multiple word segment vectors corresponding to the multiple word segments of the translated text respectively. Here, i < N, and N is the total number of layers of the pre-trained language model. According to experimental experience, it can be selected denotes rounding down to an integer. In this way, the word segment vectors of the training data combination are calculated in the intermediate layer of the pre-trained language model, and then the similarity loss value between the vector representations of the training data combination is calculated based on the word segment vectors of the training data combination. It should be understood that the smaller the similarity loss value of the training data combination, the higher the prediction similarity of the pre-trained language model for the training data combination. That is to say, text semantic information independent of language can be learned in the lower layer of the pre-trained language model, and knowledge of specific downstream tasks can be learned in the upper layer of the pre-trained language model. This hierarchical learning method greatly improves the generalization ability of the pre-trained language model and also reduces the risk of overfitting of the pre-trained language model to a certain extent.

[0080] In some embodiments, the vector similarity between the vector representation of the original text and the vector representation of the translated text can be calculated to measure the similarity loss value between the vector representation of the original text and the vector representation of the translated text. In one embodiment, the calculation method of the similarity loss value between the vector representation of the original text and the vector representation of the translated text includes: calculating the mean of multiple word segmentation vectors corresponding to multiple word segmentations of the original text as the text vector of the original text; calculating the mean of multiple word segmentation vectors corresponding to multiple word segmentations of the translated text as the text vector of the translated text; calculating the vector similarity between the text vector of the original text and the text vector of the translated text, and calculating the similarity loss value based on the vector similarity. The greater the vector similarity, the closer the similarity loss value is to 0. In one embodiment, the calculation formula of the text vector of the original text is as follows:

[0081]

[0082] Among them, S1 is the text vector of the original text of the training sample corpus, n1 is the number of word segments in the original text of the training sample corpus, is the output of the jth word segmentation of the original text of the training sample corpus in the i-th layer of the pre-trained language model, 0≤j≤n1.

[0083] In one embodiment, the calculation formula of the text vector of the translated text is as follows:

[0084]

[0085] Among them, S2 is the text vector of the translation text of the training sample corpus, n2 is the number of word segments in the translation text of the training sample corpus, is the output of the jth word segmentation of the translation text of the training sample corpus in the i-th layer of the pre-trained language model, 0≤j≤n2.

[0086] In one embodiment, the calculation formula of the similarity loss value is:

[0087]

[0088] Among them, loss is the similarity loss value between the original text and the translated text of the corpus of the training sample, S1 is the text vector of the original text of the corpus of the training sample, S2 is the text vector of the translated text of the corpus of the training sample, cos(S1, S2) is the cosine similarity between the text vector of the original text and the text vector of the translated text of the corpus of the training sample, sim_label is the label, sim_label=1 means that the original text and the translated text of the corpus of the training sample are positive samples, 0°≤the angle between the text vector of the original text and the text vector of the translated text<90°, sim_label=0 means that the original text and the translated text of the corpus of the training sample are negative samples, 90°≤the angle between the text vector of the original text and the text vector of the translated text≤180°.

[0089] It should be understood that when calculating the loss value of the pre-trained language model, the similarity loss value is added, so that the pre-trained language model can combine the training data that express the same semantics in different languages ​​into a similar semantic space during the fine-tuning process, thereby improving the cross-language representation ability of the model. In this way, in multilingual scenarios, most text understanding tasks focus more on the semantics of the text. If the text vector representation is only related to semantics and not much to language, then for low-resource languages, there is no need for training samples of specific downstream tasks. It is only necessary to provide a large number of training samples of high-resource languages ​​for specific downstream tasks and fine-tune the pre-trained language model to obtain a natural language processing model that supports low-resource languages ​​and is suitable for processing specific downstream tasks, thereby improving the performance of the natural language processing model in processing downstream tasks in low-resource language scenarios, and also increasing the range of downstream tasks that the pre-trained language model can be applied to in multilingual scenarios.

[0090] In step S230, the model parameters of the pre-trained language model are updated using the loss value to improve the similarity of the output data combination.

[0091] In some embodiments, before using the loss value to update the model parameters of the pre-trained language model, a linear weighted sum of the similarity loss value, the prediction loss value of the original text of the corpus of the training sample, and the prediction loss value of the translated text of the corpus of the training sample is calculated as the loss value of the training data combination of the training sample. In some embodiments, the loss value of the training data combination can be implemented by a calculator implemented by software and / or hardware, for example Figure 3 Adder 340. In some embodiments, updating the model parameters of the pre-trained language model using the loss value can be implemented by a parameter updater implemented by software and / or hardware, such as Figure 3 Parameter update module 330.

[0092] In some embodiments, the training data combination includes multiple bilingual data pairs. The training sample set used for fine-tuning the pre-trained language model here includes a large number of training samples, and each training sample needs to execute the method of step S210 and step S220 to obtain the loss value of the bilingual data pair of each training sample. Multiple training sample data can be packaged into a batch, such as 32 training sample data are packaged into a whole, and then the mean of the loss values ​​of multiple bilingual data pairs in the entire batch is used to update the model parameters of the pre-trained language model using a deep learning optimization algorithm. Since the use of a deep learning optimization algorithm to update the model parameters of the pre-trained language model is a prior art, it is not repeated. In this way, the parallel computing power of the GPU is fully utilized (a whole batch of data can be processed at one time), and the fine-tuning efficiency of the pre-trained language model is improved. After adjusting the model parameters, the prediction accuracy of the original text, the prediction accuracy of the translated text, and the similarity between the model prediction value of the original text and the model prediction value of the translated text of the pre-trained language model are all improved. After multiple iterations of fine-tuning training, a natural language processing model can be generated that supports more languages ​​and achieves good natural language processing results on specific downstream tasks.

[0093] Commercial value of this disclosure

[0094] The disclosed embodiments improve the fine-tuning efficiency of pre-trained language models that support multiple languages. After successful fine-tuning of the pre-trained language model, a natural language processing model that supports more languages ​​and achieves good natural language processing effects on specific downstream tasks is generated, which improves the performance of the natural language processing model in processing downstream tasks in low-resource language scenarios and the types and range of downstream tasks that the pre-trained language model can be applied to in multi-language scenarios, and has a broad application space and market value.

[0095] Training device for pre-trained language model according to an embodiment of the present disclosure

[0096] According to an embodiment of the present disclosure, a training device for a pre-trained language model is also provided. Figure 1 The model fine-tuning device 112 is shown. Figure 3 As shown, the training device of the pre-trained language model may include:

[0097] The training data combination acquisition unit 310 is used to obtain training data combinations that express the same semantics in different languages.

[0098] The loss value calculation unit 320 is used to input the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model obtains an output data combination corresponding to the training data combination after pre-training the training data combination, and calculates the loss value of the training data combination based on the output data combination.

[0099] The model parameter updating unit 330 is used to update the model parameters of the pre-trained language model using the loss value to improve the similarity of the output data combination.

[0100] Since the implementation details of the training device for the above-mentioned pre-trained language model have been described in the detailed introduction of the method embodiment above, they will not be repeated here for the sake of saving space.

[0101] Computer device according to an embodiment of the present disclosure

[0102] According to an embodiment of the present disclosure, the training method of the pre-trained language model can be performed by Figure 5 The computer device 800 can be implemented as Figure 1 The model fine-tuning device 112 is provided. Figure 5 The computer device 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0103] like Figure 5 As shown, the computer device 800 is in the form of a general computing device. The components of the computer device 800 may include but are not limited to: at least one processing unit 810, at least one storage unit 820, and a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810).

[0104] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 executes the steps of various exemplary embodiments of the present disclosure described in the description of the exemplary method above. For example, the processing unit 810 can execute the following steps: Figure 2 The steps shown in .

[0105] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0106] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0107] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0108] The computer device 800 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 800, and / or communicate with any device that enables the computer device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the computer device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the computer device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0109] It should be understood that the above is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, there are many variations of the embodiments of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0110] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0111] It should be understood that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] It should be understood that an element described in singular form herein or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown as separate herein may be combined into a single module or element, and modules or elements described or shown as single herein may be split into multiple modules or elements.

[0113] It should also be understood that the terms and expressions used herein are for description only, and one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or parts thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, changes and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. A method for training a pre-trained language model, comprising: Obtain a combination of training data that express the same semantics in different languages; Inputting the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model obtains an output data combination corresponding to the training data combination after pre-training the training data combination, and calculates a loss value of the training data combination according to the output data combination, wherein the pre-trained language model is used to represent a natural language processing model that is obtained by fine-tuning the training samples of the first resource language and is capable of processing downstream tasks of the second resource language, and the amount of data resources of the first resource language is greater than the amount of data resources of the second resource language; The loss value is used to update the model parameters of the pre-trained language model to improve the similarity of the output data combination.

2. The training method according to claim 1, wherein: The training data combination at least includes a first text expressing a first semantic meaning in a first language and a second text expressing the first semantic meaning in a second language, the first text includes a plurality of first segmented words, and the second text includes a plurality of second segmented words. Inputting the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model pre-trains the training data combination to obtain an output data combination corresponding to the training data combination, and calculating the loss value of the training data combination according to the output data combination includes: Inputting the plurality of first word segmentations into the pre-trained language model, and obtaining a plurality of first word segmentation vectors from the pre-trained language model; Inputting the plurality of second word segmentations into the pre-trained language model, and obtaining a plurality of second word segmentation vectors from the pre-trained language model; Based on the multiple first word segmentation vectors and the multiple second word segmentation vectors, a similarity loss value between the vector representation of the first text and the vector representation of the second text is calculated.

3. The training method according to claim 2, wherein: Inputting the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model pre-trains the training data combination to obtain an output data combination corresponding to the training data combination, further comprising: Inputting the first text into the pre-trained language model, and obtaining a first text prediction value from the pre-trained language model; Inputting the second text into the pre-trained language model, and obtaining a second text prediction value from the pre-trained language model; Calculating the loss value of the training data combination according to the output data combination also includes: Calculating a prediction loss value of the first text based on the first text prediction value and the label value of the first text; Based on the second text prediction value and the label value of the second text, a prediction loss value of the second text is calculated.

4. The training method according to claim 3, wherein: Before updating the model parameters of the pre-trained language model using the loss value, the method further includes: A linear weighted sum of the similarity loss value, the prediction loss value of the first text, and the prediction loss value of the second text is calculated as the loss value of the training data combination.

5. The training method according to claim 4, wherein: The training data combination includes multiple bilingual data pairs. Updating the model parameters of the pre-trained language model using the loss value includes: Using the mean of the loss values of the multiple bilingual data pairs and updating the model parameters of the pre-trained language model by using a deep learning optimization algorithm.

6. The training method according to claim 2, wherein: Calculating the similarity loss value between the vector representation of the first text and the vector representation of the second text based on the multiple first token vectors and the multiple second token vectors includes: Calculating the mean of the multiple first token vectors as the text vector of the first text; Calculating the mean of the multiple second token vectors as the text vector of the second text; Calculating the vector similarity between the text vector of the first text and the text vector of the second text, and calculating the similarity loss value based on the vector similarity.

7. The training method according to claim 2, wherein: Inputting the multiple first tokens into the pre-trained language model and obtaining multiple first token vectors by the pre-trained language model includes: Inputting the multiple first tokens into the pre-trained language model, and outputting the multiple first token vectors by the i-th layer of the pre-trained language model; Inputting the multiple second tokens into the pre-trained language model and obtaining multiple second token vectors by the pre-trained language model includes: Inputting the multiple second tokens into the pre-trained language model, and outputting the multiple second token vectors by the i-th layer of the pre-trained language model, where i < N and N is the total number of layers of the pre-trained language model.

8. The training method according to claim 2, wherein: Obtaining the training data combination expressing the same semantics in different languages includes: Obtaining the translated text of the training sample; Performing tokenization on the original text and the translated text of the training sample to obtain the training data combination expressing the same semantics in different languages.

9. A training device for a pre-trained language model, comprising: A training data combination acquisition unit, configured to obtain a training data combination expressing the same semantics in different languages; A loss value calculation unit, configured to input the training data combination into a pre-trained language model with different language understanding capabilities, so that the pre-trained language model performs pre-training on the training data combination to obtain an output data combination corresponding to the training data combination, and calculates the loss value of the training data combination according to the output data combination; A model parameter update unit, configured to update the model parameters of the pre-trained language model by using the loss value to improve the similarity of the output data combination, where the pre-trained language model is used to represent a natural language processing model obtained by fine-tuning the training samples of the first resource language and capable of processing downstream tasks of the second resource language, and the data resource volume of the first resource language is greater than the data resource volume of the second resource language.

10. A computer device, comprising: A memory, configured to store computer-executable code; A processor, configured to execute the computer-executable code to implement the method according to any one of claims 1-8.

11. A computer readable medium, characterized in that Including computer-executable code, which when executed by a processor implements the method according to any one of claims 1-8.

Images