Training methods, language processing methods and devices for pre-trained language models

Abstract

The present disclosure provides a training method, a language processing method and a device for a pre-trained language model, and relates to the field of artificial intelligence technology. The implementation scheme is: obtain a pre-training data set; wherein the pre-training data set includes: single language data of at least one language and at least one set of parallel language data of two languages, the parallel language data includes first language data and second language data, there is a corresponding relationship between the first language data and the second language data, the language includes natural language and programming language, the pre-training language model is trained according to the single language data, and the pre-training language model is trained according to the first language data and the second language data to generate a trained pre-trained language model. Thus, the generated trained pre-trained language model can handle the understanding and generation tasks across natural languages ​​and / or programming languages, and can also handle the understanding and generation tasks of natural languages ​​and / or programming languages.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a training method for a pre-trained language model, a language processing method and a device. Background Art

[0002] In the related art, the pre-training model of the programming language can be pre-trained using the open source code data of the programming language to handle the understanding and generation tasks of the programming language.

[0003] However, pre-trained models for programming languages ​​do not support understanding and generation tasks across programming languages ​​and / or natural languages, which is an urgent problem that needs to be solved. Summary of the invention

[0004] The present disclosure provides a training method, a language processing method and a device for a pre-trained language model to generate a trained pre-trained language model that can handle understanding and generation tasks across natural languages ​​and / or programming languages, and can also handle understanding and generation tasks of natural languages ​​and / or programming languages.

[0005] According to one aspect of the present disclosure, a method for training a pre-trained language model is provided, comprising: obtaining a pre-training data set; wherein the pre-training data set comprises: single language data of at least one language and at least one set of parallel language data of two languages, the parallel language data comprises first language data and second language data, there is a corresponding relationship between the first language data and the second language data, the language comprises a natural language and a programming language, the pre-trained language model is trained according to the single language data, and the pre-trained language model is trained according to the first language data and the second language data, and a trained pre-trained language model is generated.

[0006] By implementing the embodiments of the present disclosure, a pre-trained data set is obtained; wherein the pre-trained data set includes: single language data of at least one language and at least one set of parallel language data of two languages, the parallel language data includes first language data and second language data, there is a corresponding relationship between the first language data and the second language data, the language includes a natural language and a programming language, the pre-trained language model is trained according to the single language data, and the pre-trained language model is trained according to the first language data and the second language data, and a trained pre-trained language model is generated. Thus, the generated trained pre-trained language model can handle cross-natural language and / or programming language understanding and generation tasks, and can also handle natural language and / or programming language understanding and generation tasks.

[0007] According to two aspects of the present disclosure, a language processing method is provided, including: obtaining a trained language processing model; wherein the trained language processing model is obtained by fine-tuning a pre-trained language model obtained by using the methods described in some of the above embodiments according to a fine-tuning training data set and a language processing instruction; obtaining data to be processed; and inputting the data to be processed and the language processing instruction into the trained language processing model to generate target prediction data.

[0008] According to a third aspect of the present disclosure, a training device for a pre-trained language model is provided, comprising: a data acquisition unit, used to acquire a pre-training data set; wherein the pre-training data set comprises: single language data of at least one language and at least one set of parallel language data of two languages, the parallel language data comprises first language data and second language data, there is a corresponding relationship between the first language data and the second language data, and the language comprises a natural language and a programming language; a model training unit, used to train the pre-trained language model according to the single language data, and to train the pre-trained language model according to the first language data and the second language data, to generate a trained pre-trained language model.

[0009] According to a fourth aspect of the present disclosure, a language processing device is provided, including: a model acquisition unit, used to acquire a trained language processing model; wherein the trained language processing model is obtained by fine-tuning a trained pre-trained language model obtained by using the methods described in some of the above embodiments according to a fine-tuning training data set and a language processing instruction; a data to be processed acquisition unit, used to acquire data to be processed; and a language processing unit, used to input the data to be processed and the language processing instruction into the trained language processing model to generate target prediction data.

[0010] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods described in the first aspect.

[0011] According to the sixth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method described in the second aspect.

[0012] According to a seventh aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the first aspect.

[0013] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the second aspect.

[0014] According to a ninth aspect of the present disclosure, a computer program product is proposed, characterized in that when an instruction processor in the computer program product executes, a method as described in any one of the first aspects above is implemented.

[0015] According to a tenth aspect of the present disclosure, a computer program product is proposed, characterized in that when an instruction processor in the computer program product executes, a method as described in any one of the second aspects above is implemented.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0018] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0019] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0020] Figure 3 is a schematic diagram of a training process provided according to the second embodiment of the present disclosure;

[0021] Figure 4 is a schematic diagram of another training process provided according to the second embodiment of the present disclosure;

[0022] Figure 5 is a schematic diagram according to a third embodiment of the present disclosure;

[0023] Figure 6 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0024] Figure 7 is another schematic diagram according to the fourth embodiment of the present disclosure;

[0025] Figure 8 is another schematic diagram according to the fourth embodiment of the present disclosure;

[0026] Fig. 9 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0027] Fig.10 It is a block diagram of an electronic device used to implement the training method or language processing method of the pre-trained language model of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] The present invention provides a method for training a pre-trained language model. Figure 1 is a schematic diagram according to the first embodiment of the present disclosure.

[0030] like Figure 1 As shown, the method includes:

[0031] S1: Obtain a pre-training data set; wherein the pre-training data set includes: single language data of at least one language and at least one set of parallel language data of two languages, the parallel language data includes first language data and second language data, there is a corresponding relationship between the first language data and the second language data, and the language includes a natural language and a programming language.

[0032] In an embodiment of the present disclosure, a pre-training data set is obtained, and the pre-training data set includes single-language data of at least one language, wherein the language includes a natural language and a programming language, that is, the pre-training data set may include single-language data of at least one natural language and / or single-language data of at least one programming language.

[0033] Exemplarily, the natural language single language data may be: English single language data, Chinese single language data, French single language data, Spanish single language data, Arabic single language data, and so on.

[0034] Exemplarily, the single-language data of a programming language may be: single-language data of C language, single-language data of C++ language, single-language data of Python language, single-language data of Java language, and so on.

[0035] It should be noted that the above examples are for illustration only. When data of the corresponding language can be obtained, the single language data of the language can be data of any natural language or any programming language (including new programming languages ​​that will appear in the future), and the embodiments of the present disclosure do not impose specific restrictions on this.

[0036] In an embodiment of the present disclosure, a pre-training data set is obtained, and the pre-training data set includes at least one set of parallel language data in two languages, wherein the languages ​​include natural languages ​​and programming languages, that is, the pre-training data set may include at least one of the following:

[0037] A set of parallel language data in at least two natural languages;

[0038] A set of parallel language data for at least two programming languages;

[0039] A set of parallel language data for at least one natural language and one programming language.

[0040] It can be understood that in the embodiment of the present disclosure, the parallel language data includes the first language data and the second language data, and there is a corresponding relationship between the first language data and the second language data.

[0041] Among them, in the case where the parallel language data in two languages ​​are parallel language data in two natural languages, the parallel language data include first language data and second language data. The first language data and the second language data have a corresponding relationship, and the first language data and the second language data can be translations of each other, the second language data can be obtained by translating the first language data, or the first language data can be obtained by translating the second language data.

[0042] Among them, in the case where the parallel language data of two languages ​​are parallel language data of two programming languages, the parallel language data include first language data and second language data. The first language data and the second language data have a corresponding relationship, and the first language data and the second language data can be translations of each other, the second language data can be compiled from the first language data, or the first language data can be compiled from the second language data.

[0043] Among them, in the case where the parallel language data of two languages ​​are parallel language data of a natural language and a programming language, the parallel language data includes first language data and second language data, and the corresponding relationship between the first language data and the second language data may be a relationship between code and comment or a relationship between comment and code, and the second language data may be obtained by processing the first language data according to preset rules, or the first language data may be obtained by processing the second language data according to preset rules.

[0044] Exemplarily, the parallel language data of two natural languages ​​may be: parallel language data of English and Chinese, parallel language data of English and French, parallel language data of Chinese and Spanish, parallel language data of Russian and Chinese, and so on.

[0045] Exemplarily, the parallel language data of two programming languages ​​may be: parallel language data of C language and C++ language, parallel language data of C language and Python language, parallel language data of Python language and Java language, and so on.

[0046] Exemplarily, the parallel language data of a natural language and a programming language may be parallel language data of English and C language, parallel language data of English and C++ language, parallel language data of Java language and Chinese language, parallel language data of Chinese and Python language, and so on.

[0047] It should be noted that the above examples are for illustration only, and when data in the corresponding languages ​​can be obtained, the parallel language data of the two languages ​​may be data of any two languages ​​of natural language and programming language, and the embodiments of the present disclosure do not impose any specific restrictions on this.

[0048] In the embodiments of the present disclosure, the data content in the pre-training data set is as described above. The methods for obtaining different data contents may be, for example, directly obtained from the Internet, or open source data may be used, or methods in related technologies may be used to obtain the data. The embodiments of the present disclosure do not impose any specific restrictions on this.

[0049] In some embodiments, single-language data of the programming languages ​​in the pre-training dataset can be obtained from open source code communities such as GitHub, including code data of 32+ programming languages.

[0050] In the disclosed embodiment, after the pre-training data set is acquired, S2 may be executed.

[0051] S2: training the pre-trained language model according to the single language data, and training the pre-trained language model according to the first language data and the second language data to generate a trained pre-trained language model.

[0052] In the disclosed embodiment, after the pre-training data set is acquired, the pre-training model may be trained according to the pre-training data set, wherein the pre-training model may be a Transformer model.

[0053] The pre-trained language model is trained based on single language data of at least one language, and the pre-trained language model can be trained based on single language data of natural language and / or based on single language data of programming language. Thus, the generated trained pre-trained language model can handle the task of understanding and generating natural language and / or programming language.

[0054] Natural language understanding tasks, such as sentiment analysis, text classification, text matching, machine reading comprehension, information extraction, named entity recognition, etc.

[0055] Programming language understanding tasks, such as clone detection, defect detection, code retrieval, code completion, etc.

[0056] Generation tasks across natural languages ​​(text-to-text), such as document translation, etc.; generation tasks across programming languages ​​(code-to-code), such as code translation, code repair, etc.; generation tasks across natural languages ​​and programming languages ​​(code-to-text, text-to-code), such as code summary generation, code comment generation, text-to-code generation, etc.

[0057] In the disclosed embodiment, the pre-trained language model is trained based on single language data of natural language.

[0058] Exemplarily, the monolingual data of a natural language is single English data, and the span or word token in the single English data can be replaced with a mask and then input into the pre-trained model to predict the complete data. By analyzing the predicted data and the single English data, the parameters of the pre-trained model are adjusted to train the pre-trained model.

[0059] In the disclosed embodiment, the pre-trained language model is trained based on the single language data of the programming language.

[0060] Exemplarily, the single language data of the programming language is single Java data, and the fragment span or word token in the single Java data can be replaced with a mask and then input into the pre-trained model to predict the complete data. By analyzing the predicted data and the single Java data, the parameters of the pre-trained model are adjusted to train the pre-trained model.

[0061] It should be noted that in the above example, the training of the pre-trained language model is only for illustration, and the method can also be to randomly arrange the sentences in the monolingual data, input the pre-trained language model, predict the complete data, and adjust the parameters of the pre-trained model by analyzing the predicted data and the monolingual data to train the pre-trained model, etc. The embodiments of the present disclosure do not impose specific restrictions on this.

[0062] The pre-trained language model is trained according to at least one set of parallel language data in two languages, including at least one of the following:

[0063] Train the pre-trained language model based on parallel language data of two natural languages;

[0064] Train the pre-trained language model based on the parallel language data of two programming languages;

[0065] The pre-trained language model is trained on parallel language data of one natural language and one programming language.

[0066] In the disclosed embodiment, the pre-trained language model is trained based on the parallel language data of two natural languages, so that the generated pre-trained language model can handle cross-natural language understanding and generation tasks.

[0067] Exemplarily, the parallel language data of two natural languages ​​includes first language data and second language data, wherein the first language data is Chinese data and the second language data is English data.

[0068] The pre-trained language model is trained based on the parallel language data of two natural languages. Chinese data can be input into the pre-trained language model to predict English data. By analyzing the English prediction data and the English data in the parallel language data, the parameters of the pre-trained model are adjusted to train the pre-trained model.

[0069] It should be noted that in the above examples, in the parallel data of two natural languages, the first language data can also be English data, and the second language data can also be Spanish data, or the first language data can also be Chinese data, and the second language data can also be Arabic data, and so on. The process of training the pre-trained model is similar, and the embodiments of the present disclosure will not go into details.

[0070] In the disclosed embodiment, the pre-trained language model is trained based on the parallel language data of two programming languages, so that the generated pre-trained language model can handle cross-programming language understanding and generation tasks.

[0071] Exemplarily, the parallel language data of two programming languages ​​include first language data and second language data, wherein the first language data is C language data and the second language data is Java language data.

[0072] The pre-trained language model is trained based on the parallel language data of the two programming languages. The C language data can be input into the pre-trained language model and predicted as the predicted data of the Java language. By analyzing the predicted data of the Java language and the Java language data in the parallel language data, the parameters of the pre-trained model are adjusted to train the pre-trained model.

[0073] It should be noted that, in the above examples, in the parallel data of the two programming languages, the first language data can also be Java language data, and the second language data can also be Python language data, or the first language data can also be C++ language, and the second language data can also be C language data, and so on. The process of training the pre-trained model is similar, and the embodiments of the present disclosure will not go into details.

[0074] In the disclosed embodiment, the pre-trained language model is trained based on parallel language data of a natural language and a programming language, so that the generated pre-trained language model can handle the understanding and generation tasks across natural languages ​​and programming languages.

[0075] Exemplarily, the parallel language data of a natural language and a programming language include first language data of the natural language and second language data of the programming language, wherein the first language data is Chinese data and the second language data is Java language data.

[0076] The pre-trained language model is trained based on parallel language data of a natural language and a programming language. Chinese data can be input into the pre-trained language model and predicted as prediction data of Java language. By analyzing the prediction data of Java language and the Java language data in the parallel language data, the parameters of the pre-trained model are adjusted to train the pre-trained model.

[0077] It should be noted that, in the above examples, in the parallel language data of a natural language and a programming language, the first language data can also be Java language data, and the second language data can also be English data, or the first language data can also be C++ language, and the second language data can also be Spanish data, and so on. The process of training the pre-trained model is similar, and the embodiments of the present disclosure will not go into details.

[0078] It should be noted that the training of the pre-trained language model is only for illustration. The method can also be to randomly arrange the sentences in the monolingual data, input the pre-trained language model, predict the complete data, and adjust the parameters of the pre-trained model by analyzing the predicted data and the monolingual data to train the pre-trained model, etc. The embodiments of the present disclosure do not impose specific restrictions on this.

[0079] In some embodiments, at least one set of parallel data in two languages ​​includes at least one set of parallel language data in a specified natural language and another natural language, and at least one set of parallel language data in a specified natural language and a programming language.

[0080] The specified natural language may be English.

[0081] In the disclosed embodiment, when the parallel data of two languages ​​includes at least one group of parallel language data of English and one other natural language, and at least one group of parallel language data of English and one programming language, the pre-trained language model is trained according to the first language data and the second language data in the parallel language data. Thus, English can be used as the pivot language, and three methods of programming language-English, English-programming language, and English-other natural languages ​​can be used for joint training, so that the trained pre-trained language model can support the execution of tasks across multiple programming languages ​​and multiple natural languages.

[0082] Of course, the designated natural language may also be other languages ​​besides English, such as Chinese, or other natural languages, and the embodiments of the present disclosure do not impose specific limitations on this.

[0083] By implementing the embodiments of the present disclosure, a pre-trained data set is obtained; wherein the pre-trained data set includes: single language data of at least one language and at least one set of parallel language data of two languages, the parallel language data includes first language data and second language data, there is a corresponding relationship between the first language data and the second language data, the language includes a natural language and a programming language, the pre-trained language model is trained according to the single language data, and the pre-trained language model is trained according to the first language data and the second language data, and a trained pre-trained language model is generated. Thus, the generated trained pre-trained language model can handle cross-natural language and / or programming language understanding and generation tasks, and can also handle natural language and / or programming language understanding and generation tasks.

[0084] Figure 2 It is a schematic diagram of the second embodiment of the present disclosure.

[0085] like Figure 2 As shown, another method for training a pre-trained language model provided in an embodiment of the present disclosure includes:

[0086] S10: Obtain a pre-training data set; wherein the pre-training data set includes: single language data of at least one language and at least one set of parallel language data of two languages, the parallel language data includes first language data and second language data, there is a corresponding relationship between the first language data and the second language data, and the language includes a natural language and a programming language.

[0087] Among them, the description of S10 in the embodiment of the present disclosure can refer to the description of S1 in the above embodiment, and will not be repeated here.

[0088] S20: Generate model input data based on the single language data.

[0089] In the embodiment of the present disclosure, the single language data may include single language data in a natural language and single language data in a programming language, wherein model input data may be generated based on the single language data in the natural language, and model input data may be generated based on the single language data in the programming language.

[0090] In some embodiments, generating model input data based on the single language data includes at least one of the following:

[0091] Replacing at least one span in the single language data with a mask to generate a first type of model input data;

[0092] Replacing at least one example token in the monolingual data with a mask to generate a second type of model input data;

[0093] Deleting at least one example token in the single language data to generate a third type of model input data;

[0094] Randomly arrange the sentences in the monolingual data to generate the fourth type of model input data;

[0095] Connect the single language data end to end and disconnect it with any word token as the beginning to generate the fifth type of model input data.

[0096] In the embodiment of the present disclosure, the first type of model input data can be generated by replacing at least one segment span in the single language data as a mask based on the single language data of a natural language and / or a programming language.

[0097] Among them, a T5-type mask language model can be used to replace at least one span segment in the single language data with a mask to generate the first type of model input data.

[0098] like Figure 3 As shown, Figure 3(a) is a schematic diagram of generating model input data (Inputs) based on single language data (Original text) of a programming language. Figure 3 (b) is a schematic diagram of generating model input data (Inputs) based on natural language monolingual data (Original text).

[0099] In the embodiment of the present disclosure, based on the single language data of natural language and / or programming language, at least one word example token in the single language data can be replaced with a mask to generate the second type of model input data.

[0100] Among them, the masking method of the BERT model can be used to randomly replace at least one (for example, 15%) word example token in the single language data as a mask to generate the second type of model input data.

[0101] Here is a brief introduction to the BERT model. The BERT model is an unsupervised natural language processing (NLP) training model. The full name of BERT is "Bidirectional Encoder Representation from Transformers", which is the bidirectional encoder Encoder of the transformer "Transformer". Its structure is the encoding part of the Transformer tool.

[0102] In the embodiment of the present disclosure, based on the single language data of the natural language and / or programming language, at least one word example token in the single language data can be deleted to generate the third type of model input data.

[0103] In the embodiment of the present disclosure, based on the monolingual data of natural language and / or programming language, the sentences in the monolingual data can be randomly arranged to generate the fourth type of model input data.

[0104] In the disclosed embodiment, based on the single language data of natural language and / or programming language, the single language data can be connected end to end and disconnected with any word token as the beginning to generate the fifth type of model input data.

[0105] It should be noted that the above method of generating model input data based on single language data can adopt the method in the relevant technology, which will not be repeated here.

[0106] Please continue to see Figure 2 , after executing S20, execute S30.

[0107] S30: Input the model input data into the pre-trained language model to generate predicted single language data.

[0108] like Figure 3 As shown, Figure 3 (a) is a schematic diagram of generating model input data (Inputs) based on monolingual data (Original text) of a programming language, and inputting the model input data into a pre-trained language model to generate predicted monolingual data (Targets). Figure 3 (b) is a schematic diagram of generating model input data (Inputs) based on natural language monolingual data (Original text), and inputting the model input data into the pre-trained language model to generate predicted monolingual data (Targets).

[0109] In the embodiment of the present disclosure, when model input data is generated based on single language data, if one type of model input data is generated based on single language data of only one language, the obtained model input data can be input into a pre-trained language model to generate predicted single language data.

[0110] In some embodiments, model input data is input into a pre-trained language model to generate predicted monolingual data, including: in response to the monolingual data, multiple types of model input data are generated, and according to the type of the model input data, the monolingual training instructions corresponding to the model input data are determined; the model input data and the monolingual training instructions corresponding to the model input data are input into the pre-trained language model to generate predicted monolingual data.

[0111] In an embodiment of the present disclosure, when multiple types of model input data are generated based on monolingual data, the monolingual training instructions corresponding to the model input data can be determined based on the type of the model input data; the model input data and the monolingual training instructions corresponding to the model input data are input into a pre-trained language model to generate predicted monolingual data.

[0112] Exemplarily, when generating a first type of model input data and a second type of model input data based on monolingual data, firstly, the monolingual training instructions corresponding to the first type of model input data and the monolingual training instructions corresponding to the second type of model input data are determined.

[0113] The monolingual training instruction corresponding to the first type of model input data may indicate the span at which the mask is generated.

[0114] The monolingual training instruction corresponding to the second type of model input data may indicate the generation of a token at the location of the mask.

[0115] Based on this, the model input data and the monolingual training instructions corresponding to the model input data can be input into the pre-trained language model to generate predicted monolingual data. Thus, the generated trained pre-trained language model can handle the task of understanding and generating natural language and / or programming language.

[0116] S40: updating parameters of the pre-trained language model according to the monolingual data and the predicted monolingual data.

[0117] In the embodiments of the present disclosure, when single language data and predicted single language data are obtained, parameters of the pre-trained language model may be updated according to the single language data and the predicted single language data.

[0118] It should be noted that when multiple types of model input data are generated based on single language data, and multiple model input data are input into a pre-trained language model to generate multiple predicted single language data, the parameters of the pre-trained language model can be updated separately based on the single language data and each type of predicted single language data.

[0119] S50: Training the pre-trained language model according to the first language data and the second language data.

[0120] In the embodiment of the present disclosure, the first language data and the second language data may both be natural languages, or may both be programming languages, or may one be a natural language and the other be a programming language.

[0121] In the embodiment of the present disclosure, in view of the fact that the first language data and the second language data are different, different processing methods are selected when training the pre-trained language model.

[0122] In some embodiments, a bilingual training instruction is determined based on the first language data and the second language data; wherein the bilingual training instruction is used to instruct to convert the first language data into the second language data and / or convert the second language data into the first language data.

[0123] Then, the first language data and the bilingual training instructions are input into the pre-trained language model to generate predicted second language data, and the parameters of the pre-trained language model are updated according to the second language data and the predicted second language data; and / or the second language data and the bilingual training instructions are input into the pre-trained language model to generate predicted first language data, and the parameters of the pre-trained language model are updated according to the first language data and the predicted first language data.

[0124] Exemplarily, when both the first language data and the second language data are natural languages, and the first language data is Russian, and the second language data is Chinese, the bilingual training instruction is determined to be a Russian-to-Chinese instruction and / or a Chinese-to-Russian instruction.

[0125] Among them, Figure 4 As shown, Figure 4 In (a), when the bilingual training instruction is a Russian-to-Chinese instruction, the first language data in Russian is input into the pre-trained language model (Inputs, input), and the predicted second language data (Targets, target) is generated, and the second language data is predicted to be Chinese. Further, according to the second language data in Chinese and the predicted second language data in Chinese, the parameters of the pre-trained language model are updated.

[0126] In the case where the bilingual training instruction is a Chinese-to-English instruction, the second language data in Chinese is input into the pre-trained language model to generate predicted first language data, and the predicted first language data is Russian. Further, the parameters of the pre-trained language model are updated based on the first language data in Russian and the predicted first language data in Russian.

[0127] Exemplarily, when one of the first language data and the second language data is a natural language and the other is a programming language, and the first language data is English and the second language data is Python, the bilingual training instruction is determined to be an English-to-Python instruction and / or a Python-to-English instruction.

[0128] Among them, Figure 4 As shown, Figure 4 In (b), when the bilingual training instruction is an English-to-Python language instruction, the first language data in English is input into the pre-trained language model (Inputs, input), and the predicted second language data (Targets, target) is generated, and the second language data is predicted to be Python language. Further, according to the second language data in Python language and the predicted second language data in Python language, the parameters of the pre-trained language model are updated.

[0129] Among them, Figure 4 As shown, Figure 4 In (c), when the bilingual training instruction is a Python-to-English instruction, the second language data in Python is input into the pre-trained language model (Inputs, input), and the predicted first language data (Targets, target) is generated, and the predicted first language data is English. Further, according to the first language data in English and the predicted first language data in English, the parameters of the pre-trained language model are updated.

[0130] It should be noted that, in the above examples, the first language data and the second language data may also be other natural languages ​​or programming languages, and the embodiments of the present disclosure do not impose specific limitations on this.

[0131] In the disclosed embodiment, a pre-trained language model is trained based on first language data and second language data, so that the trained pre-trained language model can handle cross-programming language and / or natural language understanding and generation tasks.

[0132] In some embodiments, at least one set of parallel data in two languages ​​includes at least one set of parallel language data in a specified natural language and another natural language, and at least one set of parallel language data in a specified natural language and a programming language.

[0133] The specified natural language may be English.

[0134] In the disclosed embodiments, the pre-trained model is trained using parallel language data of English and multiple programming languages, as well as parallel language data of English and multiple other natural languages. English can be used as the hub language and joint training can be performed using three methods: programming language-English, English-programming language, and English-other natural languages. This allows the trained pre-trained language model to support the execution of tasks across multiple programming languages ​​and multiple natural languages.

[0135] Of course, the designated natural language may also be other languages ​​besides English, such as Chinese, or other natural languages, and the embodiments of the present disclosure do not impose specific limitations on this.

[0136] By implementing the embodiments of the present disclosure, a pre-trained data set is obtained, model input data is generated based on single language data, the model input data is input into a pre-trained language model, predicted single language data is generated, parameters of the pre-trained language model are updated based on the single language data and the predicted single language data, and the pre-trained language model is trained based on the first language data and the second language data. Thus, the generated trained pre-trained language model can handle cross-natural language and / or programming language understanding and generation tasks, and can also handle natural language and / or programming language understanding and generation tasks.

[0137] Figure 5 It is a schematic diagram of the third embodiment of the present disclosure.

[0138] like Figure 5 As shown, a language processing method provided by an embodiment of the present disclosure includes:

[0139] S100: Obtain a trained language processing model; wherein the trained language processing model is obtained by fine-tuning a pre-trained language model obtained by using a training method for a pre-trained language model as provided in some of the above embodiments according to a fine-tuning training data set and a language processing instruction.

[0140] S200: Obtain data to be processed.

[0141] S300: Input the data to be processed and the language processing instructions into the trained language processing model to generate target prediction data.

[0142] In the embodiments of the present disclosure, when a trained pre-trained language model is obtained by adopting the training method of the pre-trained language model provided in some of the above embodiments, the trained pre-trained language model can be fine-tuned to perform downstream language processing tasks.

[0143] Among them, the downstream tasks that can be performed are related to the fine-tuning training dataset used by the fine-tuned pre-trained language model.

[0144] In the disclosed embodiment, a corresponding fine-tuning training data set may be obtained and a language processing instruction may be determined according to the requirements of downstream tasks.

[0145] Exemplarily, the downstream task requirement is to convert the to-be-processed data in English into target data in Chinese. At this time, the fine-tuning training data set obtained includes English training samples and corresponding Chinese training samples, and the language processing instructions can be determined as English-to-Chinese instructions.

[0146] Furthermore, the trained pre-trained language model can be fine-tuned using the English training samples and the corresponding Chinese training samples to obtain a trained language processing model. Afterwards, the trained language processing model can be applied in language processing.

[0147] Exemplarily, the downstream task requirement is to convert the to-be-processed data in Python language into target data in Chinese. At this time, the fine-tuning training data set obtained includes Python language training samples and corresponding Chinese training samples, and the language processing instructions can be determined as Python-to-Chinese instructions.

[0148] Furthermore, the trained pre-trained language model can be fine-tuned using the Python language training samples and the corresponding Chinese training samples to obtain a trained language processing model. Afterwards, the trained language processing model can be applied in language processing.

[0149] It should be noted that the above examples are for illustration only, and the requirements of downstream tasks also include other types besides the above examples, such as: information extraction, text classification, code filling, etc., and the embodiments of the present disclosure do not impose specific restrictions on this.

[0150] It can be understood that in the embodiments of the present disclosure, when a trained language processing model is obtained based on a fine-tuning training data set and language processing instructions, in the application of the trained language processing model, the data to be processed is related to the fine-tuning training data set, and the same language processing instructions as used during fine-tuning are used.

[0151] By implementing the disclosed embodiments, a trained language processing model is obtained; wherein the trained language processing model is obtained by fine-tuning a pre-trained language model obtained by using the training method of the pre-trained language model provided in some of the above embodiments according to the fine-tuning training data set and the language processing instruction, obtaining the data to be processed, inputting the data to be processed and the language processing instruction into the trained language processing model, and generating target prediction data. Thus, the trained language processing model can handle specific cross-natural language and / or programming language understanding and generation tasks, and / or handle specific natural language and / or programming language understanding and generation tasks.

[0152] Figure 6 It is a schematic diagram of the fourth embodiment of the present disclosure.

[0153] like Figure 6 As shown, the fourth embodiment of the present disclosure further provides a training device 1 for a pre-trained language model, wherein the training device 1 for a pre-trained language model includes: a data acquisition unit 11 and a model training unit 12.

[0154] The data acquisition unit 11 is used to acquire a pre-training data set; wherein the pre-training data set includes: single language data of at least one language and at least one set of parallel language data of two languages, the parallel language data includes first language data and second language data, there is a corresponding relationship between the first language data and the second language data, and the language includes a natural language and a programming language.

[0155] The model training unit 12 is used to train the pre-trained language model according to the single language data, and to train the pre-trained language model according to the first language data and the second language data, so as to generate a trained pre-trained language model.

[0156] like Figure 7 As shown, in some embodiments, the model training unit 12 includes: a data processing module 121, a data prediction module 122 and a first parameter updating module 123.

[0157] The data processing module 121 is used to generate model input data according to the single language data.

[0158] The data prediction module 122 is used to input the model input data into the pre-trained language model to generate predicted single language data.

[0159] The first parameter updating module 123 is used to update the parameters of the pre-trained language model according to the monolingual data and the predicted monolingual data.

[0160] In some embodiments, the data processing module 121 is configured to perform at least one of the following:

[0161] Replacing at least one span in the single language data with a mask to generate a first type of model input data;

[0162] Replacing at least one example token in the monolingual data with a mask to generate a second type of model input data;

[0163] Deleting at least one example token in the single language data to generate a third type of model input data;

[0164] Randomly arrange the sentences in the monolingual data to generate the fourth type of model input data;

[0165] Connect the single language data end to end and disconnect it with any word token as the beginning to generate the fifth type of model input data.

[0166] In some embodiments, the data prediction module 122 is specifically configured to:

[0167] In response to generating multiple types of model input data according to the monolingual data, determining the monolingual training instructions corresponding to the model input data according to the types of the model input data;

[0168] The model input data and the monolingual training instructions corresponding to the model input data are input into the pre-trained language model to generate predicted monolingual data.

[0169] like Figure 8 As shown, in some embodiments, the model training unit 12 includes: an instruction determination module 124 and a second parameter updating module 125.

[0170] The instruction determination module 124 is used to determine the bilingual training instruction according to the first language data and the second language data; wherein the bilingual training instruction is used to instruct to convert the first language data into the second language data and / or convert the second language data into the first language data;

[0171] The second parameter updating module 125 is used to input the first language data and the bilingual training instruction into the pre-trained language model, generate predicted second language data, and update the parameters of the pre-trained language model according to the second language data and the predicted second language data; and / or input the second language data and the bilingual training instruction into the pre-trained language model, generate predicted first language data, and update the parameters of the pre-trained language model according to the first language data and the predicted first language data.

[0172] It should be noted that the aforementioned explanation of the training method of the pre-trained language model is also applicable to the training device of the pre-trained language model of this embodiment, and will not be repeated here.

[0173] The beneficial effects achieved by the training device for the pre-trained language model provided in the embodiments of the present disclosure are the same as the beneficial effects achieved by the training method for the pre-trained language model provided in the above embodiments, and will not be repeated here.

[0174] Fig. 9 Schematic diagram of the fifth embodiment of the present disclosure.

[0175] like Fig. 9 As shown, the fifth embodiment of the present disclosure further provides a language processing device 10 , wherein the language processing device 10 includes: a model acquisition unit 101 , a to-be-processed data acquisition unit 102 , and a language processing unit 103 .

[0176] The model acquisition unit 101 is used to acquire a trained language processing model; wherein the trained language processing model is obtained by fine-tuning a pre-trained language model obtained by using a training method for a pre-trained language model as in some of the above embodiments according to a fine-tuning training data set and a language processing instruction.

[0177] The to-be-processed data acquisition unit 102 is used to acquire the to-be-processed data.

[0178] The language processing unit 103 is used to input the data to be processed and the language processing instructions into the trained language processing model to generate target prediction data.

[0179] It should be noted that the above explanation of the language processing method is also applicable to the language processing device of this embodiment and will not be repeated here.

[0180] The beneficial effects achieved by the language processing device provided in the embodiments of the present disclosure are the same as the beneficial effects achieved by the language processing method provided in the above embodiments, and will not be repeated here.

[0181] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0182] Fig.10 It is a block diagram of an electronic device used to implement the training method or language processing method of the pre-trained language model of the embodiment of the present disclosure.

[0183] Electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0184] like Fig.10 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0185] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0186] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, for example, a training method or a language processing method for a pre-trained language model.

[0187] For example, in some embodiments, the training method or language processing method of the pre-trained language model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the training method or language processing method of the pre-trained language model described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the training method or language processing method of the pre-trained language model by any other appropriate means (e.g., by means of firmware).

[0188] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0189] The program code for implementing the training method or language processing method of the pre-trained language model of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0190] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0191] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0192] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0193] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0194] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as an open, inclusive meaning, that is, "including, but not limited to". In the description of the specification, the terms "some embodiments", "exemplary embodiments", "examples", etc. are intended to indicate that specific features, structures, materials or characteristics related to the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics described may be included in any one or more embodiments or examples in any appropriate manner.

[0195] Unless otherwise specified, "plurality" means two or more. The terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0196] The use of "for" herein is meant to be open and inclusive language that does not exclude devices adapted or configured to perform additional tasks or steps.

[0197] Additionally, the use of “based on” is meant to be open and inclusive, as a process, step, calculation, or other action “based on” one or more stated conditions or values ​​may, in practice, be based on additional conditions or values ​​beyond those stated.

[0198] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this disclosure can be achieved, and this document is not limited here.

[0199] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a pre-trained language model, comprising: Acquire a pre-training data set; wherein the pre-training data set includes: single language data of at least one language and at least one group of parallel language data of two languages, the parallel language data includes first language data and second language data, there is a corresponding relationship between the first language data and the second language data, the language includes a natural language and a programming language, wherein the at least one group of parallel language data of two languages ​​includes at least one group of parallel language data of a specified natural language and another natural language, and / or a group of parallel language data of a specified natural language and a programming language, wherein the corresponding relationship between the natural language and the programming language is the relationship between comments and codes; the single language data of at least one language includes single language data of at least one natural language and / or single language data of a programming language; Training a pre-trained language model according to the single language data, and training the pre-trained language model according to the first language data and the second language data to generate a trained pre-trained language model; The step of training the pre-trained language model according to the first language data and the second language data includes: Determining a bilingual training instruction according to the first language data and the second language data; wherein the bilingual training instruction is used to instruct to convert the first language data into the second language data and / or to convert the second language data into the first language data; Inputting the first language data and the bilingual training instruction into the pre-trained language model to generate predicted second language data, and updating parameters of the pre-trained language model according to the second language data and the predicted second language data; Inputting the second language data and the bilingual training instruction into the pre-trained language model to generate predicted first language data, and updating parameters of the pre-trained language model according to the first language data and the predicted first language data; The step of training the pre-trained language model according to the single language data includes: In response to generating multiple types of model input data according to the single language data, determining the monolingual training instructions corresponding to the model input data according to the types of the model input data; Inputting the model input data and the monolingual training instructions corresponding to the model input data into the pre-trained language model to generate predicted monolingual data; Parameters of the pre-trained language model are updated according to the single language data and the predicted single language data.

2. The method according to claim 1, wherein: The generating of multiple types of model input data according to the single language data includes at least one of the following: Replacing at least one span in the single language data with a mask to generate the model input data of the first type; Replacing at least one example token in the single language data with a mask to generate a second type of the model input data; Deleting at least one example token in the single language data to generate the model input data of the third type; Randomly arrange the sentences in the monolingual data to generate a fourth type of the model input data; The single language data are connected end to end and separated by taking any word token as the beginning to generate the fifth type of the model input data.

3. A language processing method, comprising: Obtaining a trained language processing model; wherein the trained language processing model is obtained by fine-tuning a pre-trained language model obtained by the method according to any one of claims 1 to 2 according to a fine-tuning training data set and a language processing instruction; Get the data to be processed; The data to be processed and the language processing instructions are input into the trained language processing model to generate target prediction data.

4. A training device for a pre-trained language model, comprising: A data acquisition unit, used to acquire a pre-training data set; wherein the pre-training data set includes: single language data of at least one language and at least one group of parallel language data of two languages, the parallel language data includes first language data and second language data, there is a corresponding relationship between the first language data and the second language data, the language includes a natural language and a programming language, wherein the at least one group of parallel language data of two languages ​​includes at least one group of parallel language data of a specified natural language and another natural language, and / or a group of parallel language data of a specified natural language and a programming language, wherein the corresponding relationship between a natural language and a programming language is the relationship between comments and codes; the single language data of at least one language includes single language data of at least one natural language and / or single language data of a programming language; A model training unit, configured to train a pre-trained language model according to the single language data, and to train the pre-trained language model according to the first language data and the second language data, to generate a trained pre-trained language model; Wherein, the model training unit includes: An instruction determination module, configured to determine a bilingual training instruction according to the first language data and the second language data; wherein the bilingual training instruction is used to instruct to convert the first language data into the second language data and / or to convert the second language data into the first language data; a second parameter updating module, configured to input the first language data and the bilingual training instruction into the pre-trained language model, generate predicted second language data, and update parameters of the pre-trained language model according to the second language data and the predicted second language data; and / or Inputting the second language data and the bilingual training instruction into the pre-trained language model to generate predicted first language data, and updating parameters of the pre-trained language model according to the first language data and the predicted first language data; Wherein, the model training unit includes: A data processing module, used for generating model input data according to the single language data; A data prediction module, configured to generate a plurality of types of the model input data in response to the monolingual data, determine the monolingual training instructions corresponding to the model input data according to the types of the model input data; input the model input data and the monolingual training instructions corresponding to the model input data into the pre-trained language model to generate predicted monolingual data; The first parameter updating module is used to update the parameters of the pre-trained language model according to the single language data and the predicted single language data.

5. The device according to claim 4, wherein: The data processing module is used to perform at least one of the following: Replacing at least one span in the single language data with a mask to generate the model input data of the first type; Replacing at least one example token in the single language data with a mask to generate a second type of the model input data; Deleting at least one example token in the single language data to generate the model input data of the third type; Randomly arrange the sentences in the monolingual data to generate a fourth type of the model input data; The single language data are connected end to end and separated by taking any word token as the beginning to generate the fifth type of the model input data.

6. A language processing device, comprising: A model acquisition unit, used to acquire a trained language processing model; wherein the trained language processing model is obtained by fine-tuning a pre-trained language model obtained by the method according to any one of claims 1 to 2 according to a fine-tuning training data set and a language processing instruction; A to-be-processed data acquisition unit, used for acquiring the to-be-processed data; The language processing unit is used to input the data to be processed and the language processing instructions into the trained language processing model to generate target prediction data.

7. An electronic device, wherein: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of any one of claims 1 to 2, or the instructions are executed by the at least one processor so that the at least one processor can execute the method of claim 3.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 2, or the computer instructions are used to cause the computer to execute the method according to claim 3.

9. A computer program product, wherein: When the instruction processor in the computer program product is executed, the method according to any one of claims 1 to 2 is implemented, or when the instruction processor in the computer program product is executed, the method according to claim 3 is implemented.

Images