Training method of pre-trained language model, training method and device of language model

Abstract

The present disclosure provides a training method for a pre-trained language model, a training method for a language model and a device, comprising: obtaining a sample text, performing at least two of the pre-training tasks according to the sample text, and obtaining a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task and a masked language modeling task, the forward causal language modeling task is a modeling task of predicting a subsequent word in the sample text by a preceding word in the sample text, the reverse causal language modeling task is a modeling task of predicting a preceding word in the sample text by a subsequent word in the sample text, and the masked language modeling task is a modeling task of predicting a word in a masked position in the sample text by a word in a non-masked position in the sample text, and pre-training can be achieved from multiple dimensions, thereby achieving diversity and flexibility in training, and the pre-trained language model obtained by training from multiple dimensions can have higher accuracy and reliability.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a training method for a pre-trained language model, a training method for a language model, and a device. Background Art

[0002] With the development of the Internet and artificial intelligence technology, pre-trained language models have been widely used in various natural language processing tasks, such as dialogue systems, machine translation, text summarization, information retrieval, and title generation.

[0003] In the related art, sample data may be collected to train a basic network model based on the sample data, thereby obtaining a pre-trained language model. Summary of the invention

[0004] The present disclosure provides a training method for a pre-trained language model, a training method for a language model, and a device for the training method, so as to improve the reliability of the pre-trained language model.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for pre-training a language model, comprising:

[0006] Get sample text;

[0007] Performing at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task, and a masked language modeling task;

[0008] Among them, the forward causal language modeling task is a modeling task of predicting the subsequent words in the sample text by the previous words in the sample text; the reverse causal language modeling task is a modeling task of predicting the previous words in the sample text by the subsequent words in the sample text; the masked language modeling task is a modeling task of predicting the words in the masked position in the sample text by the words in the non-masked position in the sample text.

[0009] In one embodiment of the present disclosure, performing at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model includes:

[0010] Performing word segmentation processing on the sample text to obtain each word in the sample text;

[0011] At least two of the pre-training tasks are performed according to each word in the sample text to obtain a pre-trained language model.

[0012] In one embodiment of the present disclosure, if the at least two pre-training tasks performed include the reverse causal language modeling task, performing the reverse causal language modeling task according to each word in the sample text includes:

[0013] For any word in the sample text, predict the preceding adjacent word of the word in the sample text according to the word and the future word;

[0014] The future word is a word located after any one of the words in the sample text; and the preceding adjacent word is a word located before any one of the words in the sample text and adjacent to any one of the words.

[0015] In one embodiment of the present disclosure, if the at least two pre-training tasks performed include the forward causal language modeling task, performing the forward causal language modeling task according to each word in the sample text includes:

[0016] For any word in the sample text, predict the subsequent adjacent word of the word in the sample text according to the word and the historical words;

[0017] The historical word is a word located before any word in the sample text; the subsequent adjacent word is a word located after any word and adjacent to any word in the sample text.

[0018] In one embodiment of the present disclosure, if the at least two pre-training tasks performed include the masked language modeling task, performing the masked language modeling task according to each word in the sample text includes:

[0019] Performing masking on at least one word in the sample text to obtain a masked position;

[0020] The word at the masked position is predicted based on the word at the non-masked position in the sample text.

[0021] In one embodiment of the present disclosure, the training modes of the at least two pre-training tasks include: an alternating training mode, a parallel training mode, and a serial training mode.

[0022] In a second aspect, the present disclosure provides a method for training a language model, including:

[0023] Obtain training text for the target task scenario;

[0024] Training a pre-trained language model according to the training text to obtain a language model suitable for the target task scenario;

[0025] The pre-trained language model is trained based on the method described in the first aspect.

[0026] In a third aspect, an embodiment of the present disclosure provides a training device for a pre-trained language model, comprising:

[0027] A first acquisition unit, used for acquiring a sample text;

[0028] An execution unit, configured to execute at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task, and a masked language modeling task;

[0029] Among them, the forward causal language modeling task is a modeling task of predicting the subsequent words in the sample text by the previous words in the sample text; the reverse causal language modeling task is a modeling task of predicting the previous words in the sample text by the subsequent words in the sample text; the masked language modeling task is a modeling task of predicting the words in the masked position in the sample text by the words in the non-masked position in the sample text.

[0030] In one embodiment of the present disclosure, the execution unit includes:

[0031] A word segmentation subunit is used to perform word segmentation processing on the sample text to obtain each word in the sample text;

[0032] The execution subunit is used to execute at least two of the pre-training tasks according to each word in the sample text to obtain a pre-trained language model.

[0033] In one embodiment of the present disclosure, if the at least two pre-training tasks performed include the reverse causal language modeling task, the execution subunit is used to, for any word in the sample text, predict the preceding adjacent word of the any word in the sample text according to the any word and the future word;

[0034] The future word is a word located after any one of the words in the sample text; and the preceding adjacent word is a word located before any one of the words in the sample text and adjacent to any one of the words.

[0035] In one embodiment of the present disclosure, if the forward causal language modeling task is included in the at least two pre-training tasks to be performed, the execution subunit is used to predict, for any word in the sample text, the subsequent adjacent word of the any word in the sample text according to the any word and the historical words;

[0036] The historical word is a word located before any word in the sample text; the subsequent adjacent word is a word located after any word and adjacent to any word in the sample text.

[0037] In one embodiment of the present disclosure, if at least two pre-training tasks are performed including the masked language modeling task, the execution subunit is used to perform mask processing on at least one word in the sample text to obtain a masked position, and predict the word at the masked position based on the word at the non-masked position in the sample text.

[0038] In one embodiment of the present disclosure, the training modes of the at least two pre-training tasks include: an alternating training mode, a parallel training mode, and a serial training mode.

[0039] In a fourth aspect, an embodiment of the present disclosure provides a language model training device, including:

[0040] A second acquisition unit is used to acquire training text of the target task scenario;

[0041] A training unit, used to train a pre-trained language model according to the training text to obtain a language model suitable for the target task scenario;

[0042] The pre-trained language model is trained based on the method described in the first aspect.

[0043] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0044] at least one processor; and

[0045] a memory communicatively connected to at least one processor; wherein,

[0046] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the electronic device to perform the method described in the first aspect or the second aspect of the present disclosure.

[0047] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect or the second aspect of the present disclosure is implemented.

[0048] In a seventh aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or the second aspect of the present disclosure.

[0049] The embodiments of the present disclosure provide a method for training a pre-trained language model, a method for training a language model, and a device. The technical feature of the pre-trained language model is obtained by combining at least two pre-training tasks among a forward causal language modeling task, a reverse causal language modeling task, and a masked language modeling task. Training from multiple dimensions can be achieved, the diversity and flexibility of pre-training can be achieved, and the pre-trained language model obtained from training in multiple dimensions can have higher accuracy and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 A flowchart of a method for training a pre-trained language model according to an embodiment of the present disclosure;

[0052] Figure 2 A flowchart of a method for training a pre-trained language model according to another embodiment of the present disclosure;

[0053] Figure 3 A schematic diagram of the principle of the forward causal language modeling task in an embodiment of the present disclosure;

[0054] Figure 4 A schematic diagram of the principle of the reverse causal language modeling task in an embodiment of the present disclosure;

[0055] Figure 5 A schematic diagram of the principle of the masked language modeling task according to an embodiment of the present disclosure;

[0056] Figure 6 A schematic diagram of a method for training a language model according to an embodiment of the present disclosure;

[0057] Figure 7 A training device for a pre-trained language model according to an embodiment of the present disclosure;

[0058] Figure 8 A training device for a pre-trained language model according to another embodiment of the present disclosure;

[0059] Fig. 9 A schematic diagram of a language model training device according to an embodiment of the present disclosure;

[0060] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure.

[0061] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0063] The terms "first", "second", "third", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in sequences other than those illustrated or described herein.

[0064] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0065] Natural Language Processing (NLP) technology refers to a technology that studies language problems in the interaction between humans and computers.

[0066] With the development of the Internet and artificial intelligence technology, pre-trained language models have been widely used in various natural language processing tasks, such as dialogue systems, machine translation, text summarization, information retrieval, and title generation.

[0067] In some embodiments, the basic network model can be trained through sample data training to obtain a pre-trained language model. The basic network model can be an unsupervised deep bidirectional (BERT) model, a generative unsupervised training (GPT) model, and the like.

[0068] In order to improve the effect of the pre-trained language model and improve the accuracy of natural language processing tasks, the inventors of the present disclosure have obtained the inventive concept of the present disclosure through creative labor: performing multiple pre-training tasks to obtain a pre-trained language model by combining multiple pre-training tasks, wherein the multiple pre-training tasks include at least two of the forward causal language modeling task, the reverse causal language modeling task, and the masked language modeling task.

[0069] The technical solution of the present disclosure is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0070] See also Figure 1 , Figure 1 Flowchart of a method for training a pre-trained language model according to an embodiment of the present disclosure. Figure 1 As shown, the method includes:

[0071] S101: Obtain sample text.

[0072] Exemplarily, the executor of this embodiment may be a training device for a pre-trained language model (hereinafter referred to as a training device), and the training device may be a server (such as a local server or a cloud server), a computer, a terminal device, a processor, a chip, etc., which is not limited in this embodiment.

[0073] The sample text may be a sentence, a paragraph, or an article, etc. The present embodiment does not limit the length of the sample text, and it may be determined based on demand, historical records, and experiments, etc. Similarly, the number of sample texts may also be determined based on demand, historical records, and experiments, etc.

[0074] For example, for scenarios with higher accuracy requirements, the number of sample texts may be relatively large, while for scenarios with lower accuracy requirements, the number of sample texts may be relatively small.

[0075] S102: Perform at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model.

[0076] Among them, the pre-training tasks include: forward causal language modeling task, reverse causal language modeling task and masked language modeling task.

[0077] The forward causal language modeling task is to predict the word after the sample text from the previous word in the sample text. The reverse causal language modeling task is to predict the word before the sample text from the word after the sample text. The masked language modeling task is to predict the word at the masked position in the sample text from the word at the non-masked position in the sample text.

[0078] Exemplarily, two of the forward causal language modeling task, the reverse causal language modeling task, and the masked language modeling task may be performed according to the sample text to train a pre-trained language model. The forward causal language modeling task, the reverse causal language modeling task, and the masked language modeling task may also be performed according to the sample text to train a pre-trained language model.

[0079] There is no limitation on executing two of the above three tasks or executing the order of the above three tasks, that is, the training modes of at least two pre-training tasks include: alternating training mode, parallel training mode and serial training mode.

[0080] Taking the alternating training mode as an example, the forward causal language modeling task can be performed first, and then the reverse causal language modeling task can be performed, and then the forward causal language modeling task can be performed. Taking the parallel training mode as an example, the forward causal language modeling task, the reverse causal language modeling task, and the mask language modeling task can be performed separately. Taking the serial training mode as an example, the forward causal language modeling task can be performed first, and then the reverse causal language modeling task can be performed.

[0081] In combination with the above analysis, it can be seen that an embodiment of the present disclosure provides a training method for a pre-trained language model, including: obtaining a sample text, performing at least two of the pre-training tasks according to the sample text, and obtaining a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task, and a masked language modeling task. The forward causal language modeling task is a modeling task of predicting a word after the sample text by a word in front of the sample text, the reverse causal language modeling task is a modeling task of predicting a word in front of the sample text by a word in the back of the sample text, and the masked language modeling task is a modeling task of predicting a word in a masked position in the sample text by a word in a non-masked position in the sample text. In this embodiment, the technical feature of obtaining a pre-trained language model by combining at least two of the forward causal language modeling tasks, the reverse causal language modeling task, and the masked language modeling task can realize pre-training from multiple dimensions, thereby realizing diversity and flexibility in training, and can make the pre-trained language model obtained from training in multiple dimensions have higher accuracy and reliability.

[0082] To enable readers to more deeply understand the implementation principle of the present disclosure, the implementation principle of the present disclosure will be elaborated in detail by taking three pre-training tasks as an example.

[0083] Please refer to Figure 2 , Figure 2 which is a flowchart of a training method for a pre-trained language model according to another embodiment of the present disclosure. As Figure 2 shown, the method includes:

[0084] S201: Obtain sample text.

[0085] It should be understood that, in order to avoid cumbersome statements, for the same technical features of this embodiment and the above embodiments, this embodiment will not be described in detail again.

[0086] In some embodiments, the sample text is unsupervised text, that is, the sample text is sample text without labels.

[0087] S202: Perform word segmentation on the sample text to obtain each word in the sample text.

[0088] This embodiment does not limit the method of word segmentation. For example, stop words in the sample text can be removed first, and then the sample text after removing the stop words can be segmented to obtain each word in the sample text. Among them, stop words are words without meaning, such as adverbs like "de" and prepositions like "zai", etc.

[0089] S203: Determine the word vector corresponding to each word in the sample text, and perform encoding processing on each word vector to obtain the encoded vector of each word vector.

[0090] Exemplarily, for each word in the sample text, according to the mapping relationship between the preset word and the index, determine the index corresponding to the word, and according to the corresponding relationship between the preset index and the word vector matrix, determine the row in the word vector matrix corresponding to the index corresponding to the word from the word vector matrix, and determine the word vector of the word based on this row. The word vector is used to represent the semantic features of the word.

[0091] For example, as Figure 3 shown, the words in the sample text include: x1, x2, x3, and x4. Input x1, x2, x3, x4, and the beginning special symbol (begin-of-sentence, BOS, used to identify the beginning of the sample text) into the embedding network, and output the word vectors corresponding to x1, x2, x3, x4, and BOS respectively. Input each word vector into the encoding network (transformer) to obtain the encoded vector corresponding to each word vector.

[0092] As Figure 3As shown in , the transformer includes a multi-head self-attention network and a feed forward network. The transformer is an N-layer encoding structure, where N is a positive integer greater than or equal to 1. Figure 3 As shown in the figure, the encoding structure of a transformer layer includes a multi-head self-attention network and a feedforward neural network. After the word vector passes through the multi-head self-attention network of the current layer, it enters the feedforward neural network of the current layer, and then enters the multi-head self-attention network of the next layer, and so on.

[0093] S204: For any word in the sample text, according to the encoding vector of the any word and the encoding vectors of the historical words, a forward prediction result of the any word is predicted.

[0094] That is, the forward prediction result of any word is a word located after and adjacent to any word in the sample text. The historical word is a word located before any word in the sample text.

[0095] Exemplarily, combining the above analysis and Figure 3 As shown, each coding vector is input into a multilayer perceptron (MLP) model, and the forward prediction result of each coding vector is output. Figure 3 As shown, the multilayer perceptron model outputs the first word x1 in the sample text according to the encoding vector of BOS; the multilayer perceptron model outputs the second word x2 in the sample text according to the encoding vector of BOS and the encoding vector of x1, and so on. The multilayer perceptron model outputs the end special symbol (end-of-sentence, EOS, used to mark the end of the sample text) according to the encoding vector of BOS, the encoding vector of x1, the encoding vector of x2, the encoding vector of x3 and the encoding vector of x4.

[0096] S205: Determine a forward pre-training loss function according to the forward prediction result, that is, the forward pre-training loss function represents a loss function based on the forward causal language modeling task.

[0097] S206: For any word in the sample text, according to the encoding vector of any word and the encoding vector of the future word, predict and obtain the reverse prediction result of the any word.

[0098] That is, the reverse prediction result of any word is a word located before and adjacent to any word in the sample text. The future word is a word located after any word in the sample text.

[0099] Exemplarily, combining the above analysis and Figure 4As shown in , each encoding vector is input into the multi-layer perceptron model, and the reverse prediction result of each encoding vector is output. Figure 4 As shown, the multilayer perceptron model outputs the last word x4 in the sample text according to the encoding vector of BOS; the multilayer perceptron model outputs the third word x3 in the sample text according to the encoding vector of BOS and the encoding vector of x4, and so on. The multilayer perceptron model outputs the ending special symbol according to the encoding vector of BOS, the encoding vector of x4, the encoding vector of x3, the encoding vector of x2, and the encoding vector of x1.

[0100] S207: Determine a reverse pre-training loss function according to the reverse prediction result, that is, the reverse pre-training loss function represents a loss function based on the reverse causal language modeling task.

[0101] S208: Mask at least one word in the sample text to obtain a mask position, and perform mask prediction on the word at the mask position according to the word at the non-mask position in the sample text to obtain a mask prediction result, wherein the mask prediction result represents the word at the mask position in the sample text.

[0102] Exemplarily, combining the above analysis and Figure 5 As shown, the last word in the sample text (i.e. x4) is masked, as Figure 5 As shown, x4 is masked and marked as mask, and each encoding vector is input into the multilayer perceptron model. The multilayer perceptron model outputs x4 based on the encoding vector of x1, the encoding vector of x2, and the encoding vector of x3.

[0103] S209: Determine a mask pre-training loss function according to the mask prediction result, that is, the mask pre-training loss function represents a loss function based on the mask language modeling task.

[0104] S210: Perform reverse parameter adjustment processing according to the forward pre-training loss function, the reverse pre-training loss function and the pre-training loss function to obtain a pre-training language model.

[0105] For example, in combination with the above analysis, the forward pre-training loss function, the reverse pre-training loss function and the average loss value of the pre-training loss function can be determined, and the parameters of the embedding network, transformer and multi-layer perceptron model can be adjusted according to the average loss value until the number of iterations is met or the average loss value is less than the preset loss threshold. Similarly, the loss threshold can be determined based on demand, historical records, experiments, etc.

[0106] Combined with the above analysis, it can be seen that the pre-trained language model can be obtained by adopting the serial training mode. In other embodiments, the parameters of each model of the forward causal language modeling task can be adjusted based on the forward pre-training loss function to meet the minimum loss function of the forward causal language modeling task before executing the reverse causal language modeling task. The parameters of each model of the reverse causal language modeling task can be adjusted based on the reverse pre-training loss function to meet the minimum loss function of the reverse causal language modeling task before executing the mask language modeling task.

[0107] If the pre-trained language model is obtained by training in parallel training mode, the forward causal language modeling task, the reverse causal language modeling task and the mask language modeling task can be performed simultaneously. The pre-trained language model can be obtained by training using the above-mentioned average loss value method, or the parameters of the corresponding models can be adjusted based on their corresponding loss functions to obtain the pre-trained language model.

[0108] If the pre-trained language model is obtained by training in an alternating training mode, the parameters of each model of the forward causal language modeling task can be adjusted based on the forward pre-training loss function. When certain requirements are met (the minimum value of the loss function is not reached), the reverse causal language modeling task can be performed, and the parameters of each model of the reverse causal language modeling task can be adjusted based on the reverse pre-training loss function. When certain requirements are met (the minimum value of the loss function is not reached), the forward causal language modeling task can be performed again.

[0109] It is worth noting that there are more combinations of forward causal language modeling tasks, reverse causal language modeling tasks, and mask language modeling tasks, which are not listed here one by one. That is, each modeling task can be an independent embodiment or a combination of embodiments, and the combination method and order of each modeling task are not limited and can be determined based on needs.

[0110] Since the training modes are diverse, the ways of training to obtain pre-trained language models are also diverse. That is, by adopting the training mode provided in this embodiment, the technical effect of the diversity and flexibility of training to obtain pre-trained language models can be improved.

[0111] Based on the above analysis, we can know that the forward causal language modeling task can predict the subsequent words based on the previous words in the sample text, that is, the forward causal language modeling task can use historical information to predict future information; the reverse causal modeling task can use the subsequent words in the sample text to predict the previous words, that is, the forward causal language modeling task can use future information to predict historical information; the masked language modeling task can predict the masked words based on the unmasked words in the sample text, that is, the masked language modeling task can use certain information to predict uncertain information. Therefore, when using Figure 2 When the pre-trained language model is trained in the embodiment shown, the pre-trained language model can learn the ability to use historical information to predict future information, the ability to use future information to predict historical information, and the ability to use certain information to predict uncertain information, thereby making the pre-trained language model have stronger and more accurate prediction capabilities.

[0112] See also Figure 6 , Figure 6 A schematic diagram of a language model training method according to an embodiment of the present disclosure.

[0113] like Figure 6 As shown, the method includes:

[0114] S601: Obtain training text for a target task scenario.

[0115] Exemplarily, the executor of this embodiment may be a training device for a language model, and the training device may be the same device as the training device for the pre-trained language model, or may be a different device, which is not limited in this embodiment.

[0116] Combined with the above analysis, it can be seen that pre-trained language models are widely used in various natural language processing tasks, such as dialogue systems, machine translation, text summarization, information retrieval, and title generation, etc. Correspondingly, the target task scenario can be a dialogue system scenario, or a machine translation scenario, text summarization scenario, information retrieval scenario, title generation scenario, etc.

[0117] Taking the scenario of a dialogue system as an example, the training text is the text in the scenario of the dialogue system, such as the dialogue text of human-computer interaction. Other scenarios are similar and are not listed here one by one.

[0118] S602: Train a pre-trained language model according to the training text to obtain a language model suitable for the target task scenario.

[0119] The pre-trained language model is trained based on the method described in any of the above embodiments.

[0120] In combination with the above embodiment, if the target task scenario is a scenario of a dialogue system, the language model may determine feedback information corresponding to the dialogue information based on the dialogue information initiated by the user.

[0121] For example, a dialog system is deployed in a smart speaker, and the dialog system includes a language model. A user can initiate a dialog message to the smart speaker to query the weather by voice or touch. The language model can determine the user's intention to query the weather, and the dialog system can determine the output feedback information based on the intention. The feedback information is used to represent the weather information.

[0122] It is worth noting that since the pre-trained language model has stronger and more accurate prediction capabilities, the language model trained based on the pre-trained model has stronger language comprehension capabilities, thereby improving the comprehension accuracy of the language model.

[0123] See also Figure 7 , Figure 7 A training device for a pre-trained language model according to an embodiment of the present disclosure, such as Figure 7 As shown, the device 700 includes:

[0124] The first acquisition unit 701 is used to acquire a sample text.

[0125] The execution unit 702 is used to perform at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task, and a masked language modeling task.

[0126] Among them, the forward causal language modeling task is a modeling task of predicting the subsequent words in the sample text by the previous words in the sample text; the reverse causal language modeling task is a modeling task of predicting the previous words in the sample text by the subsequent words in the sample text; the masked language modeling task is a modeling task of predicting the words in the masked position in the sample text by the words in the non-masked position in the sample text.

[0127] See also Figure 8 , Figure 8 A training device for a pre-trained language model according to another embodiment of the present disclosure, such as Figure 8 As shown, the device 800 includes:

[0128] The first acquisition unit 801 is used to acquire a sample text.

[0129] The execution unit 802 is used to perform at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task, and a masked language modeling task.

[0130] Among them, the forward causal language modeling task is a modeling task of predicting the subsequent words in the sample text by the previous words in the sample text; the reverse causal language modeling task is a modeling task of predicting the previous words in the sample text by the subsequent words in the sample text; the masked language modeling task is a modeling task of predicting the words in the masked position in the sample text by the words in the non-masked position in the sample text.

[0131] Combination Figure 8It can be seen that, in some embodiments, the execution unit 802 includes:

[0132] The word segmentation subunit 8021 is used to perform word segmentation processing on the sample text to obtain each word in the sample text.

[0133] The execution subunit 8022 is used to perform at least two of the pre-training tasks according to each word in the sample text to obtain a pre-trained language model.

[0134] In one embodiment of the present disclosure, if at least two pre-training tasks performed include the reverse causal language modeling task, the execution subunit 8022 is used to predict, for any word in the sample text, the preceding adjacent word of the any word in the sample text based on the any word and the future word.

[0135] The future word is a word located after any one of the words in the sample text; and the preceding adjacent word is a word located before any one of the words in the sample text and adjacent to any one of the words.

[0136] In one embodiment of the present disclosure, if at least two pre-training tasks are performed including the forward causal language modeling task, the execution subunit 8022 is used to predict, for any word in the sample text, the subsequent adjacent words of the any word in the sample text based on the any word and historical words.

[0137] The historical word is a word located before any word in the sample text; the subsequent adjacent word is a word located after any word and adjacent to any word in the sample text.

[0138] In one embodiment of the present disclosure, if at least two pre-training tasks are performed including the masked language modeling task, the execution subunit 8022 is used to perform mask processing on at least one word in the sample text to obtain a masked position, and predict the word at the masked position based on the word at the non-masked position in the sample text.

[0139] In one embodiment of the present disclosure, the training modes of the at least two pre-training tasks include: an alternating training mode, a parallel training mode, and a serial training mode.

[0140] See also Fig. 9 , Fig. 9 Schematic diagram of a language model training device according to an embodiment of the present disclosure, Fig. 9 As shown, the device 900 includes:

[0141] The second acquisition unit 901 is used to acquire the training text of the target task scenario.

[0142] The training unit 902 is used to train a pre-trained language model according to the training text to obtain a language model suitable for the target task scenario.

[0143] The pre-trained language model is trained based on the method described in the first aspect.

[0144] Fig.10 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present disclosure. Fig.10 As shown, the electronic device 1000 of the embodiment of the present disclosure may include: at least one processor 1001 ( Fig.10 Only one processor is shown); and a memory 1002 in communication with at least one processor. The memory 1002 stores instructions that can be executed by at least one processor 1001, and the instructions are executed by at least one processor 1001 so that the electronic device 1000 can execute the technical solution in any of the aforementioned method embodiments.

[0145] Optionally, the memory 1002 may be independent or integrated with the processor 1001 .

[0146] When the memory 1002 is a device independent of the processor 1001 , the electronic device 1000 further includes: a bus 1003 for connecting the memory 1002 and the processor 1001 .

[0147] The electronic device provided by the embodiments of the present disclosure can execute the technical solution of any of the aforementioned method embodiments, and its implementation principles and technical effects are similar, which will not be repeated here.

[0148] The embodiments of the present disclosure further provide a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the technical solution in any of the aforementioned method embodiments.

[0149] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements the technical solution in any of the aforementioned method embodiments when executed by a processor.

[0150] The embodiment of the present disclosure further provides a chip, including: a processing module and a communication interface, wherein the processing module can execute the technical solution in the aforementioned method embodiment.

[0151] Furthermore, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution in the aforementioned method embodiment.

[0152] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0153] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0154] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present disclosure is not limited to only one bus or one type of bus.

[0155] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0156] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for training a pre-trained language model, comprising: Get sample text; Performing at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task, and a masked language modeling task, and the pre-trained language model is obtained by reverse parameter adjustment according to the forward pre-training loss function, the reverse pre-training loss function, and the loss values ​​of at least two of the pre-training loss functions; The forward causal language modeling task is a modeling task of predicting the following word in the sample text from the preceding word in the sample text; the reverse causal language modeling task is a modeling task of predicting the preceding word in the sample text from the following word in the sample text; the masked language modeling task is a modeling task of predicting the word in the masked position in the sample text from the word in the non-masked position in the sample text; The training modes of the at least two pre-training tasks include: an alternating training mode, a parallel training mode and a serial training mode.

2. The method according to claim 1, wherein: The performing at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model includes: Performing word segmentation processing on the sample text to obtain each word in the sample text; At least two of the pre-training tasks are performed according to each word in the sample text to obtain a pre-trained language model.

3. The method according to claim 2, wherein: If the at least two pre-training tasks to be performed include the reverse causal language modeling task, performing the reverse causal language modeling task according to each word in the sample text includes: For any word in the sample text, predict the preceding adjacent word of the word in the sample text according to the word and the future word; The future word is a word located after any one of the words in the sample text; and the preceding adjacent word is a word located before any one of the words in the sample text and adjacent to any one of the words.

4. The method according to claim 2 or 3, wherein: If the at least two pre-training tasks to be performed include the forward causal language modeling task, performing the forward causal language modeling task according to each word in the sample text includes: For any word in the sample text, predict the subsequent adjacent word of the word in the sample text according to the word and the historical words; The historical word is a word located before any word in the sample text; the subsequent adjacent word is a word located after any word and adjacent to any word in the sample text.

5. The method according to any one of claims 2 to 4, wherein: If the at least two pre-training tasks to be performed include the masked language modeling task, performing the masked language modeling task according to each word in the sample text includes: Performing masking on at least one word in the sample text to obtain a masked position; The word at the masked position is predicted based on the word at the non-masked position in the sample text.

6. A method for training a language model, comprising: Obtain training text for the target task scenario; Training a pre-trained language model according to the training text to obtain a language model suitable for the target task scenario; The pre-trained language model is trained based on the method described in any one of claims 1 to 5.

7. A training device for a pre-trained language model, comprising: A first acquisition unit, used for acquiring a sample text; An execution unit is used to execute at least two of the pre-training tasks according to the sample text to obtain a pre-trained language model, wherein the pre-training tasks include: a forward causal language modeling task, a reverse causal language modeling task, and a masked language modeling task, and the pre-trained language model is obtained by reverse parameter adjustment according to the forward pre-training loss function, the reverse pre-training loss function, and the loss values ​​of at least two of the pre-training loss functions; Among them, the forward causal language modeling task is a modeling task of predicting the subsequent word in the sample text by the previous word in the sample text; the reverse causal language modeling task is a modeling task of predicting the previous word in the sample text by the subsequent word in the sample text; the masked language modeling task is a modeling task of predicting the word in the masked position in the sample text by the word in the non-masked position in the sample text; the training modes of the at least two pre-training tasks include: alternating training mode, parallel training mode and serial training mode.

8. A language model training device, comprising: A second acquisition unit is used to acquire training text of the target task scenario; A training unit, used to train a pre-trained language model according to the training text to obtain a language model suitable for the target task scenario; The pre-trained language model is trained based on the method described in any one of claims 1 to 5.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the electronic device can execute any one of the methods of claims 1-5; or, the instructions are executed by the at least one processor so that the electronic device can execute the method of claim 6.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented; or, when the computer program is executed by a processor, the method according to claim 6 is implemented.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented; or, when the computer program is executed by a processor, the method according to claim 6 is implemented.

Images