Language model training method and apparatus, device, and storage medium

Abstract

This application provides a language model training method, apparatus, device, and storage medium, relating to the field of machine learning technology. The method includes: performing full pre-training on an initial student language model based on a full pre-training corpus to obtain a fully pre-trained student language model; performing incremental pre-training on the fully pre-trained student language model based on incremental pre-training corpus to obtain an incrementally pre-trained student language model; wherein the incremental pre-training corpus is additional pre-training corpus added on top of the full pre-training corpus; performing knowledge distillation on the incrementally pre-trained student language model using a teacher language model to obtain a distilled student language model; and using a second training sample set of the target task to perform parameter adjustment on the distilled student language model to obtain a fully trained student language model. Using the technical solution provided in this application can save time in language model training.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a language model training method, apparatus, device, and storage medium. Background Technology

[0002] With the development of machine learning technology, various language models have emerged for predicting language or text.

[0003] In related technologies, the model is trained using the full pre-training corpus. After the student language model is trained and the full pre-trained corpus is obtained, if new pre-training corpus (i.e. incremental pre-training corpus) is obtained, the new pre-training corpus will be added to the original full pre-training corpus to obtain an updated full pre-training corpus. Then, the student language model is retrained using the updated full pre-training corpus.

[0004] Among the aforementioned related technologies, training is time-consuming due to the large amount of data in the updated full pre-training corpus. Summary of the Invention

[0005] This application provides a language model training method, apparatus, device, and storage medium, which can save time in language model training. The technical solution is as follows:

[0006] According to one aspect of the embodiments of this application, a language model training method is provided, the method comprising:

[0007] The initial student language model is fully pre-trained based on the full pre-training corpus to obtain the fully pre-trained student language model.

[0008] The student language model after full pre-training is incrementally pre-trained based on the incremental pre-training corpus to obtain the incrementally pre-trained student language model; wherein, the incremental pre-training corpus is a pre-training corpus added on the basis of the full pre-training corpus.

[0009] The incrementally pre-trained student language model is subjected to knowledge distillation using the teacher language model to obtain the distilled student language model; wherein, the teacher language model is a language model trained using the first training sample set of the target task;

[0010] Using the second training sample set of the target task, the parameters of the distilled student language model are adjusted to obtain a fully trained student language model.

[0011] According to one aspect of the embodiments of this application, a language model training apparatus is provided, the apparatus comprising:

[0012] The full pre-training module is used to perform full pre-training on the initial student language model based on the full pre-training corpus, so as to obtain the fully pre-trained student language model.

[0013] The incremental pre-training module is used to perform incremental pre-training on the fully pre-trained student language model based on the incremental pre-training corpus to obtain the incrementally pre-trained student language model; wherein, the incremental pre-training corpus is a pre-training corpus added on the basis of the fully pre-training corpus.

[0014] The model distillation module is used to perform knowledge distillation on the incrementally pre-trained student language model using the teacher language model to obtain the distilled student language model; wherein, the teacher language model is a language model trained using the first training sample set of the target task;

[0015] The parameter adjustment module is used to adjust the parameters of the distilled student language model using the second training sample set of the target task, so as to obtain a trained student language model.

[0016] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the above-described language model training method.

[0017] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored in the computer-readable storage medium, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the above-described language model training method.

[0018] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0019] After obtaining the incremental pre-training corpus, the student language model is incrementally pre-trained using only the incremental pre-training corpus, based on the fully pre-trained student language model. The original fully pre-trained corpus does not need to participate in the incremental pre-training process, saving incremental pre-training time and thus saving the overall training time of the student language model.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a language model training method provided in one embodiment of this application;

[0023] Figure 2 This is a schematic diagram of a model structure provided in one embodiment of this application;

[0024] Figure 3 This is a flowchart of a language model training method provided in another embodiment of this application;

[0025] Figure 4 This is a flowchart of a language model training method provided in another embodiment of this application;

[0026] Figure 5 This is a schematic diagram of a sample annotation method provided in one embodiment of this application;

[0027] Figure 6 This is a flowchart of a language model training method provided in another embodiment of this application;

[0028] Figure 7 This is a block diagram of a language model training apparatus provided in one embodiment of this application;

[0029] Figure 8 This is a block diagram of a language model training apparatus provided in another embodiment of this application;

[0030] Figure 9 This is a block diagram of a terminal provided in one embodiment of this application. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods consistent with some aspects of this application as detailed in the appended claims.

[0032] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0033] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0034] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0035] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0036] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0037] The solutions provided in this application involve artificial intelligence natural language processing and machine learning technologies, such as using natural language processing technologies for word segmentation, determining the semantic features of text, and predicting text content.

[0038] The method provided in this application can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. This computer device can be a terminal such as a PC (Personal Computer), tablet computer, smartphone, wearable device, or intelligent robot; or it can be a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0039] The technical solution of this application will be described and illustrated below through several embodiments.

[0040] Please refer to Figure 1 This document illustrates a flowchart of a language model training method provided in one embodiment of this application. In this embodiment, the method is primarily illustrated by its application to the computer device described above. The method may include the following steps (101-104):

[0041] Step 101: Perform full pre-training on the initial student language model based on the full pre-training corpus to obtain the fully pre-trained student language model.

[0042] The full pre-training corpus refers to all corpora that can be used for pre-training. In this embodiment, the full pre-training corpus is used as training samples to perform full pre-training on the initial student language model, thereby obtaining the fully pre-trained student language model.

[0043] In some embodiments, step 101 further includes the following sub-steps:

[0044] 1. The full pre-training corpus is segmented to obtain multiple full pre-training corpus segments.

[0045] Optionally, the maximum length of the full pre-training corpus segment is less than or equal to a first preset length. The first preset length can be set by relevant technical personnel according to actual conditions, and this embodiment does not specifically limit it.

[0046] 2. For the j-th segment of the full pre-training corpus, cover some segments of the j-th segment of the full pre-training corpus, randomly replace some segments of the j-th segment of the full pre-training corpus with other text, and leave the remaining segments unchanged. j is a positive integer.

[0047] For example, in the j-th full pre-training corpus segment, a1% of the words are randomly selected. Of these a1% words, b1% are masked, c1% are randomly replaced with other words, and the remaining words remain unchanged. Here, a1, b1, and c1 are positive numbers, and their sum is 100. For example, a1 is 10, b1 is 80, and c1 is 10. Optionally, the specific values ​​of a1, b1, and c1 can be set by those skilled in the art based on actual conditions; this embodiment does not limit this.

[0048] 3. Map each corpus segment in the j-th full pre-training corpus segment to a semantic vector and a position vector corresponding to each corpus segment; wherein, the semantic vector is used to represent the semantic features corresponding to the corpus segment, and the position vector is used to represent the position features corresponding to the corpus segment.

[0049] 4. For each corpus segment in the j-th full pre-training corpus, add their respective semantic vectors and position vectors to obtain the vector sum corresponding to each corpus segment in the j-th full pre-training corpus, and input it into the student language model in the full pre-training for forward computation.

[0050] 5. The vector sums corresponding to each corpus segment in the j-th full pre-training corpus are input into the student language model in the full pre-training corpus, and mapped to a vector space of vocabulary size through the fully connected prediction layer of the student language model in the full pre-training corpus. This yields the prediction probability distributions corresponding to the masked text in the j-th full pre-training corpus, and the corresponding loss is determined.

[0051] In some embodiments, a third loss is determined based on the label information of the full pre-training corpus and the third prediction result of the student language model in the full pre-training corpus on the full pre-training corpus. The third loss is related to the degree of difference between the third prediction result and the label information of the full pre-training corpus. Optionally, the third prediction result includes the full predicted probability distribution corresponding to the masked text, the label information of the full pre-training corpus is the actual probability distribution of the masked text in the full pre-training corpus, and the third loss is the cross-entropy between the full predicted probability distribution and the actual probability distribution of the masked text in the full pre-training corpus.

[0052] In one example, the third loss is positively correlated with the degree of difference between the third prediction result and the label information of the full pre-training corpus. That is, the greater the difference between the third prediction result and the label information of the full pre-training corpus, the greater the third loss; the smaller the difference between the third prediction result and the label information of the full pre-training corpus, the smaller the third loss.

[0053] In another example, the third loss is negatively correlated with the degree of difference between the third prediction and the label information of the full pre-training corpus. That is, the greater the difference between the third prediction and the label information of the full pre-training corpus, the smaller the third loss; the smaller the difference between the third prediction and the label information of the full pre-training corpus, the greater the third loss.

[0054] 6. Calculate the loss corresponding to the j-th full pre-training corpus segment, and the gradient of the student language model parameters in the full pre-training corpus. Adjust the parameters of the student language model in the full pre-training corpus based on the gradient value.

[0055] 7. Using other full pre-training corpus segments, repeat steps 1 to 6 above until the third stopping training condition is met, and obtain the full pre-trained student language model.

[0056] Optionally, the third stop training condition includes: the number of parameter updates of the student language model in full pre-training reaches a target preset number, the prediction accuracy of the student language model in full pre-training reaches a target preset accuracy, etc. The specific value of the target preset number can be set by relevant technical personnel according to actual conditions, and this application embodiment does not impose specific limitations on it; the target preset accuracy can also be set by relevant technical personnel according to actual conditions, and this application embodiment does not impose specific limitations on it.

[0057] The above is merely an exemplary description of the third stop training condition. The third stop training condition may also include other conditions, which are not specifically limited in this application embodiment.

[0058] In some embodiments, such as Figure 2As shown, the student language model 20 includes a multi-head attention layer 21, a Transformer (convolutional network) layer 22, and a hidden layer (not shown in the figure). The outputs of the multi-head attention layer 21, the Transformer layer 22, and the hidden layer are the hidden layer state data, which are the intermediate outputs of the student language model 20.

[0059] Step 102: Perform incremental pre-training on the fully pre-trained student language model based on the incremental pre-training corpus to obtain the incrementally pre-trained student language model.

[0060] The incremental pre-training corpus is a new pre-training corpus added on top of the full pre-training corpus.

[0061] In some embodiments, the initial model of the student language model in incremental pre-training is the fully pre-trained student language model. After obtaining the fully pre-trained student language model through training on the full pre-training corpus, and then acquiring incremental pre-training corpus, it is necessary to further train the fully pre-trained student language model based on the incremental pre-training corpus. In this embodiment, only the incremental pre-training corpus is used to further perform incremental pre-training based on the fully pre-trained student language model. The original full pre-training corpus does not need to participate in the incremental pre-training process, saving incremental pre-training time and thus saving the overall training time of the student language model.

[0062] Step 103: Use the teacher's language model to perform knowledge distillation on the incrementally pre-trained student language model to obtain the distilled student language model.

[0063] The teacher language model is a language model trained using the first training sample set of the target task.

[0064] In some embodiments, the teacher language model has the same or similar functionality as the student language model. In one example, both the teacher and student language models are used to predict words in text. For example, both predict words in texts of the same language (e.g., Chinese, English, Japanese, German, French, etc.); both predict words in texts of the same type (e.g., news texts, novels, encyclopedic texts, etc.); and both predict words in texts related to the same field (e.g., medicine, biology, physics, chemistry, law, etc.). Optionally, the student language model has fewer parameters than the teacher language model; and / or, the student language model has fewer network layers than the teacher language model.

[0065] Optionally, the training samples in the first training sample set are labeled data for the target task; correspondingly, the teacher language model trained based on the first training sample set, and the student language model obtained based on the teacher language model, are used to perform natural language processing on the target task. For example, if the target task is to predict missing or ambiguous words in a text, then the text labeled with the correct words in the missing or ambiguous parts is used as the training samples in the first training sample set.

[0066] In some embodiments, after obtaining the incrementally pre-trained student language model through training with incremental pre-training corpus, the intermediate or final output of the teacher language model is used to influence the adjustment of the parameters of the incrementally pre-trained student language model. This can also be understood as using the teacher language model to teach the student language model, thereby transferring the "knowledge" about the target task learned by the teacher language model to the student language model, thus achieving knowledge distillation and obtaining a distilled student language model.

[0067] Step 104: Using the second training sample set of the target task, the parameters of the distilled student language model are adjusted to obtain the trained student language model.

[0068] In the previous step 104, the student language model indirectly learned the relevant knowledge of the target task through the teacher language model. Optionally, in this step 104, the parameters of the distilled student language model are directly adjusted using the second training sample set of the target task, so that the student language model directly learns the relevant knowledge of the target task, resulting in a fully trained student language model.

[0069] In summary, the technical solution provided in this application embodiment, after obtaining the incremental pre-training corpus, uses only the incremental pre-training corpus to perform incremental pre-training on the student language model based on the fully pre-trained student language model. The original fully pre-trained corpus does not need to participate in the incremental pre-training process, saving incremental pre-training time and thus saving the overall training time of the student language model.

[0070] In addition, in this embodiment, the teacher language model is used to perform knowledge distillation on the incrementally pre-trained student language model, and the teacher language model is a language model trained using the first training sample set of the target task. This saves model training time while ensuring the accuracy of the distilled student language model for the target task.

[0071] In addition, in this embodiment, the second training sample set of the target task is used to adjust the parameters of the distilled student language model, so that the student language model can directly learn the relevant knowledge of the target task, thereby further improving the accuracy of the trained student language model for the target task.

[0072] Please refer to Figure 3 This document illustrates a flowchart of a language model training method according to another embodiment of this application. In this embodiment, the method is primarily illustrated by its application to the computer device described above. The method may include the following steps (301-309):

[0073] Step 301: Perform full pre-training on the initial student language model based on the full pre-training corpus to obtain the fully pre-trained student language model.

[0074] This step 301 is related to the above. Figure 1 The content of step 101 in the embodiment is the same or similar, and will not be repeated here.

[0075] Step 302: Obtain the first prediction result and the second prediction result.

[0076] Optionally, the first prediction result refers to the prediction result of the student language model in incremental pre-training on the incremental pre-training corpus, and the second prediction result refers to the prediction result of the student language model after full pre-training on the incremental pre-training corpus. By inputting the processed incremental pre-training corpus into the student language model in incremental pre-training, the prediction result of the student language model in incremental pre-training on the incremental pre-training corpus is obtained; by inputting the processed incremental pre-training corpus into the student language model after full pre-training, the prediction result of the student language model after full pre-training on the incremental pre-training corpus is obtained.

[0077] Optionally, the prediction results are the probability distribution of the model's predictions for the specified words. The first prediction result is the probability distribution of the student language model's predictions for the specified words in the incremental pre-training corpus during incremental pre-training; the second prediction result is the probability distribution of the student language model's predictions for the specified words in the incremental pre-training corpus after full pre-training.

[0078] In some embodiments, step 302 includes the following sub-steps:

[0079] 1. Mask the target text in the incremental pre-training corpus to generate masked incremental pre-training corpus.

[0080] In some embodiments, a masked incremental pre-training corpus is used to incrementally pre-train the student language model in incremental pre-training, thereby testing the predictive ability of the student language model in incremental pre-training on the masked incremental pre-training corpus. Optionally, the target text is randomly selected and relatively dispersed text.

[0081] In some possible implementations, this sub-step may also include:

[0082] (1) The incremental pre-training corpus is segmented to obtain multiple incremental pre-training corpus segments.

[0083] Optionally, the maximum length of the incremental pre-training corpus segment is less than or equal to a second preset length. The second preset length can be set by relevant technical personnel according to actual conditions, and this embodiment does not specifically limit it.

[0084] (2) For the mth incremental pre-training corpus segment in the incremental pre-training corpus, cover some corpus segments in the mth incremental pre-training corpus segment, randomly replace some corpus segments in the mth full pre-training corpus segment with other text, and leave the remaining corpus segments unchanged, where m is a positive number.

[0085] For example, in the j-th full pre-training corpus segment, a2% of the words are randomly selected. Of these a2% words, b2% are masked, c2% are randomly replaced with other words, and the remaining words remain unchanged. Here, a2, b2, and c2 are positive numbers, and their sum is 100. For example, a2 is 10, b2 is 80, and c2 is 10. Optionally, the specific values ​​of a2, b2, and c2 can be set by those skilled in the art based on actual conditions; this embodiment does not limit this.

[0086] 2. The masked incremental pre-training corpus is processed using the student language model in the incremental pre-training to obtain the first prediction result.

[0087] Optionally, the first prediction result includes the masked text predicted by the student language model during incremental pre-training.

[0088] In some embodiments, the sub-step further includes:

[0089] (1) Map each corpus segment in the m-th incremental pre-training corpus segment to a semantic vector and a position vector corresponding to each corpus segment respectively; wherein, the semantic vector is used to represent the semantic features corresponding to the corpus segment, and the position vector is used to represent the position features corresponding to the corpus segment.

[0090] (2) For each corpus segment in the m-th incremental pre-training corpus segment, add their respective semantic vectors and position vectors to obtain the vector sum corresponding to each corpus segment in the m-th incremental pre-training corpus segment, and input it into the student language model in the incremental pre-training for forward calculation to obtain the first prediction result.

[0091] 3. The masked incremental pre-training corpus is processed by the fully pre-trained student language model to obtain the second prediction result, which includes the masked text predicted by the fully pre-trained student language model.

[0092] The specific method for obtaining the second prediction result can be found in the above description of obtaining the first prediction result, and will not be repeated here.

[0093] Step 303: Determine the first loss based on the first prediction result, the second prediction result, and the label information of the incremental pre-training corpus.

[0094] Optionally, the first loss is related to the degree of difference between the first prediction result and the label information of the incremental pre-training corpus, and also to the degree of difference between the first prediction result and the second prediction result. The first loss can be determined by integrating the first prediction result, the second prediction result, and the label information of the incremental pre-training corpus.

[0095] In some embodiments, step 303 further includes the following sub-steps:

[0096] 1. Based on the first prediction result and the label information of the incremental pre-training corpus, determine the first sub-loss. The first sub-loss is related to the degree of difference between the first prediction result and the label information of the incremental pre-training corpus.

[0097] In one example, the first sub-loss is positively correlated with the degree of difference between the first prediction and the label information of the incremental pre-training corpus. That is, the greater the difference between the first prediction and the label information of the incremental pre-training corpus, the greater the first sub-loss; the smaller the difference between the first prediction and the label information of the incremental pre-training corpus, the smaller the first sub-loss.

[0098] In one example, the first sub-loss is negatively correlated with the degree of difference between the first prediction and the label information of the incremental pre-training corpus. That is, the greater the difference between the first prediction and the label information of the incremental pre-training corpus, the smaller the first sub-loss; the smaller the difference between the first prediction and the label information of the incremental pre-training corpus, the larger the first sub-loss.

[0099] Optionally, as shown in step 302 above, the first prediction result is the prediction probability distribution of the student language model in the incremental pre-training corpus for predicting specified words in the incremental pre-training corpus. Therefore, the first prediction result corresponds to a first probability distribution vector representing its probability distribution. The label information of the incremental pre-training corpus can also be represented as a label vector of the probability distribution. Optionally, the first sub-loss is the cross-entropy between the first probability distribution vector and the label vector of the probability distribution.

[0100] In one example, the first sub-loss is calculated using the following formula:

[0101] Formula 1:

[0102] in, For the first loss, O S′ Let O be the first probability distribution vector, and let O be the label vector of the probability distribution.

[0103] 2. Based on the first and second prediction results, determine the second sub-loss, which is related to the degree of difference between the first and second prediction results.

[0104] In one example, the second sub-loss is positively correlated with the degree of difference between the first and second predictions. That is, the greater the difference between the first and second predictions, the greater the second sub-loss; the smaller the difference between the first and second predictions, the smaller the second sub-loss.

[0105] In another example, the second sub-loss is negatively correlated with the degree of difference between the first and second predictions. That is, the greater the difference between the first and second predictions, the smaller the second sub-loss; the smaller the difference between the first and second predictions, the larger the second sub-loss.

[0106] Optionally, as shown in step 302 above, the second prediction result is the prediction probability distribution of the student language model after full pre-training, predicting specified words in the incremental pre-training corpus. The second prediction result corresponds to a second probability distribution vector used to represent its probability distribution. Optionally, the second sub-loss is the cross-entropy between the first probability distribution vector and the second probability distribution vector.

[0107] In one example, the second sub-loss is calculated using the following formula:

[0108] Formula 2:

[0109] in, For the first loss, O S′ Let O be the first probability distribution vector. S This is the second probability distribution vector.

[0110] 3. Determine the first loss based on the first sub-loss and the second sub-loss.

[0111] In other embodiments, the method further includes the following steps:

[0112] 1. Obtain the first hidden state data and the second hidden state data. The first hidden state data refers to the hidden state data of the student language model in incremental pre-training when generating the first prediction result, and the second hidden state data refers to the hidden state data of the student language model after full pre-training when generating the second prediction result.

[0113] 2. Based on the first hidden layer state data and the second hidden layer state data, determine the third sub-loss. The third sub-loss is related to the degree of difference between the first hidden layer state data and the second hidden layer state data.

[0114] In some embodiments, the third sub-loss is determined by calculating the mean square error between the first hidden layer state data and the second hidden layer state data.

[0115] In one example, the third sub-loss is positively correlated with the degree of difference between the first and second hidden state data. That is, the greater the difference between the first and second hidden state data, the larger the third sub-loss; the smaller the difference between the first and second hidden state data, the smaller the third sub-loss.

[0116] In another example, the third sub-loss is positively correlated with the degree of difference between the first and second hidden state data. That is, the greater the difference between the first and second hidden state data, the smaller the third sub-loss; the smaller the difference between the first and second hidden state data, the larger the third sub-loss.

[0117] Optionally, the calculation method for the third sub-loss can refer to the following formula three:

[0118] Formula 3:

[0119] Among them, H S H represents the hidden variables of the fully pre-trained student language model when generating the second prediction result. S′ A represents the hidden variables of the student language model during incremental pre-training when generating the first prediction result; S A represents the attention weights of the fully pre-trained student language model when generating the second prediction result. S′ E represents the attention weights of the student language model during incremental pre-training when generating the first prediction result; S E represents the semantic vector matrix of the fully pre-trained student language model when generating the second prediction result. S′ This is the semantic vector matrix of the student language model during incremental pre-training when generating the first prediction result.

[0120] Accordingly, the first loss is determined based on the first sub-loss and the second sub-loss, including: determining the first loss based on the first sub-loss, the second sub-loss and the third sub-loss.

[0121] In some embodiments, determining the first loss based on the first sub-loss, the second sub-loss, and the third sub-loss includes the following steps:

[0122] 1. Obtain the weight values ​​corresponding to the first sub-loss, the second sub-loss, and the third sub-loss respectively;

[0123] 2. The first sub-loss, the second sub-loss, and the third sub-loss are weighted and summed based on their weight values ​​to obtain the first loss.

[0124] Optionally, the calculation method for the first loss can refer to the following formula four:

[0125] Formula 4:

[0126] Where, λ MLM λ is the weight value of the first sub-loss. pred λhi is the weight value of the second sub-loss. idn This is the weight value for the third sub-loss.

[0127] In some embodiments, the weight values ​​corresponding to the first sub-loss, the second sub-loss, and the third sub-loss are set by relevant technical personnel according to the actual situation, and this application embodiment does not limit this.

[0128] Step 304: Adjust the parameters of the student language model in the incremental pre-training based on the first loss until the first stopping condition is met, and obtain the student language model after incremental pre-training.

[0129] In some embodiments, after determining the first loss, the parameters of the student language model in incremental pre-training are adjusted according to the value of the first loss, and then the student language model in incremental pre-training is trained and the parameters are adjusted in the next round; until the first stopping condition is met, training is stopped, and the student language model after incremental pre-training is obtained.

[0130] Optionally, the first stopping training condition includes: the number of parameter updates of the student language model in incremental pre-training reaches a first preset number, the prediction accuracy of the student language model in incremental pre-training reaches a first preset accuracy, etc. The specific value of the first preset number can be set by relevant technical personnel according to actual conditions, and this application embodiment does not impose a specific limitation on it; the first preset accuracy can be set by relevant technical personnel according to actual conditions, and this application embodiment does not impose a specific limitation on it.

[0131] The above is merely an exemplary description of the first stop training condition. The first stop training condition may also include other conditions, which are not specifically limited in this application embodiment.

[0132] Step 305, as follows Figure 4 As shown, the first label information 43 is determined based on the teacher language model 41 and the knowledge distillation corpus 42.

[0133] In some embodiments, the knowledge distillation corpus 42 is input into the teacher language model 41 to obtain first label information 43. The first label information 43 refers to the label information obtained by the teacher language model 41 through reasoning on the knowledge distillation corpus 42. Optionally, the label information is soft label information, which refers to the feature map generated by the model during reasoning on the knowledge distillation corpus 42. Therefore, the knowledge distillation corpus 42 is a corpus that does not require manual annotation.

[0134] Soft labels are a type of labeling relative to hard labels, which are commonly used in manual annotation. In this application, the teacher's language model and / or student's language model, when predicting the text "Hong Kong remains the world's freest economy," outputs the probability distribution of each word's label. For example, the probability distribution for "Hong Kong" is [0.1, 0, ..., 0, 0.89, 0, ..., 0, 0.01], with the label having the highest probability being the final annotation result. This distribution is obtained by performing a softmax (logistic regression) operation on the score output by the neural network. The soft label in this technical solution is the smoothed neural network output, i.e., softmax(Z / T), where Z is the score output by the neural network, and T is a hyperparameter controlling the degree of smoothing.

[0135] Optionally, the knowledge distillation corpus 42 is corpus data selected from the full pre-training corpus and / or incremental pre-training corpus.

[0136] Step 306: Based on multiple incrementally pre-trained student language models 44 and knowledge distillation corpus 42, determine multiple second label information 45.

[0137] In some embodiments, the knowledge distillation corpus 42 is input into multiple incrementally pre-trained student language models 44 to obtain second label information 45 corresponding to each of the multiple incrementally pre-trained student language models 44. The second label information 45 refers to the label information obtained by the incrementally pre-trained student language models 44 through reasoning on the knowledge distillation corpus 42.

[0138] Optionally, the complexity of the multiple incrementally pre-trained student language models 44 may differ. For example, the parameters and / or structure of the multiple incrementally pre-trained student language models 44 may differ. In one example, the different structures of the multiple incrementally pre-trained student language models 44 refer to the different number of layers in the multiple incrementally pre-trained student language models 44.

[0139] In one example, the teacher language model 41 has 24 Transformer layers, 1024 hidden layer dimensions, and 16 multi-head attention layers; while the incrementally pre-trained student language model 44 can have 12 Transformer layers, 512 hidden layer dimensions, and 8 multi-head attention layers.

[0140] Step 307: For the i-th incrementally pre-trained student language model 44, based on the first label information 43, the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44, and the second label information 45 corresponding to other incrementally pre-trained student language models 44, determine the second loss 46 corresponding to the i-th incrementally pre-trained student language model 44.

[0141] In some embodiments, the second loss 46 corresponding to the i-th incrementally pre-trained student language model 44 is related to the degree of difference between the first label information 43 and the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44, and is also related to the degree of difference between the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44 and the second label information 45 corresponding to other incrementally pre-trained student language models 44, where i is a positive integer.

[0142] Optionally, the second loss 46 corresponding to the i-th incremental pre-trained student language model 44 is determined by calculating the cross-entropy between the first label information 43 and the second label information 45 corresponding to the i-th incremental pre-trained student language model 44, and the KL divergence (relative entropy) between the second label information 45 corresponding to the i-th incremental pre-trained student language model 44 and the second label information 45 corresponding to other incremental pre-trained student language models 44.

[0143] In some embodiments, step 307 further includes the following sub-steps:

[0144] 1. Based on the first label information 43 and the second label information 45 corresponding to the i-th incremental pre-trained student language model 44, determine the fourth sub-loss.

[0145] The fourth sub-loss is related to the degree of difference between the first label information 43 and the second label information 45 corresponding to the i-th incremental pre-trained student language model 44.

[0146] In one example, the fourth sub-loss is positively correlated with the degree of difference between the first label information 43 and the second label information 45 corresponding to the student language model 44 after the i-th incremental pre-training. That is, the greater the difference between the first label information 43 and the second label information 45 corresponding to the student language model 44 after the i-th incremental pre-training, the larger the fourth sub-loss; the smaller the difference between the first label information 43 and the second label information 45 corresponding to the student language model 44 after the i-th incremental pre-training, the smaller the fourth sub-loss.

[0147] In another example, the fourth sub-loss is negatively correlated with the degree of difference between the first label information 43 and the second label information 45 corresponding to the student language model 44 after the i-th incremental pre-training. That is, the greater the difference between the first label information 43 and the second label information 45 corresponding to the student language model 44 after the i-th incremental pre-training, the smaller the fourth sub-loss; the smaller the difference between the first label information 43 and the second label information 45 corresponding to the student language model 44 after the i-th incremental pre-training, the larger the fourth sub-loss.

[0148] 2. Based on the second label information 45 corresponding to the i-th incremental pre-trained student language model 44 and the second label information 45 corresponding to the other incremental pre-trained student language model 44, determine the fifth sub-loss. The fifth sub-loss is related to the degree of difference between the second label information 45 corresponding to the i-th incremental pre-trained student language model 44 and the second label information 45 corresponding to the other incremental pre-trained student language model 44.

[0149] In one example, the fifth sub-loss is positively correlated with the degree of difference between the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44 and the second label information 45 corresponding to other incrementally pre-trained student language models 44. That is, the greater the difference between the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44 and the second label information 45 corresponding to other incrementally pre-trained student language models 44, the larger the fifth sub-loss; the smaller the difference between the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44 and the second label information 45 corresponding to other incrementally pre-trained student language models 44, the smaller the fifth sub-loss.

[0150] In another example, the fifth sub-loss is negatively correlated with the degree of difference between the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44 and the second label information 45 corresponding to other incrementally pre-trained student language models 44. That is, the greater the difference between the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44 and the second label information 45 corresponding to other incrementally pre-trained student language models 44, the smaller the fifth sub-loss; the smaller the difference between the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44 and the second label information 45 corresponding to other incrementally pre-trained student language models 44, the larger the fifth sub-loss.

[0151] In some possible implementations, this step includes:

[0152] (1) Determine the KL divergence between the second label information 45 corresponding to the i-th incremental pre-trained student language model 44 and the second label information 45 corresponding to other incremental pre-trained student language models 44.

[0153] (2) Determine the fifth sub-loss based on KL divergence.

[0154] In this implementation, the fifth sub-loss is determined by calculating the KL divergence, so that the performance of each incrementally pre-trained student language model 44 is as close as possible after knowledge distillation, thereby improving the accuracy of each student language model.

[0155] 3. Based on the fourth and fifth sub-losses, determine the second loss 46 corresponding to the i-th incremental pre-trained student language model 44.

[0156] Optionally, the calculation method for the second loss 46 corresponding to the i-th incrementally pre-trained student language model 44 is as follows: Formula 5

[0157] Formula 5:

[0158] Where, p T For the first tag information 43, p i For the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44, CrossEntropy(p T ,p i p is the cross-entropy between the first label information 43 and the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44. k The second label information 45 corresponding to the other incrementally pre-trained student language models 44. Let be the KL divergence between the second label information 45 corresponding to the i-th incrementally pre-trained student language model 44 and the second label information 45 corresponding to other incrementally pre-trained student language models 44.

[0159] Step 308: Based on the second loss 46 corresponding to the i-th incrementally pre-trained student language model 44, adjust the parameters of the i-th incrementally pre-trained student language model 44 until the second stopping condition is met, and obtain the i-th distilled student language model.

[0160] In some embodiments, after determining the second loss 46, the parameters of the i-th incrementally pre-trained student language model 44 are adjusted according to the value of the second loss 46, and then the i-th incrementally pre-trained student language model 44 is trained and its parameters are adjusted in the next round until the second stopping condition is met, and then training is stopped to obtain the i-th distilled student language model.

[0161] Optionally, the second training termination condition includes: the number of parameter updates of the student language model 44 after the i-th incremental pre-training reaches a second preset number, the prediction accuracy of the student language model 44 after the i-th incremental pre-training reaches a second preset accuracy, etc. The specific value of the second preset number can be set by relevant technical personnel according to actual conditions, and this embodiment does not specifically limit it; the second preset accuracy can be set by relevant technical personnel according to actual conditions, and this embodiment does not specifically limit it.

[0162] The above is merely an exemplary description of the second training stop condition. The second training stop condition may also include other conditions, which are not specifically limited in this embodiment.

[0163] Step 309: Using the second training sample set of the target task, the parameters of the distilled student language model are adjusted to obtain the trained student language model.

[0164] This step 309 is related to the above. Figure 1 The content of step 104 in the embodiment is the same or similar, and will not be repeated here.

[0165] In summary, the technical solution provided in this application embodiment determines the fifth sub-loss by calculating the KL divergence, so that the performance of each incrementally pre-trained student language model 44 is as close as possible after knowledge distillation, thereby improving the accuracy of each student language model and enhancing the generalization performance of the student language model. The required student language model can be flexibly selected to perform the target task according to actual needs.

[0166] In addition, in this embodiment of the application, before using the teacher language model to perform knowledge distillation on the incremental student language model, the third training sample set of the target task is used to further fine-tune the teacher language model control model, thereby improving the accuracy of the teacher language model, and further improving the accuracy of the distilled student language model obtained therefrom.

[0167] In some embodiments, the method further includes: using a third training sample set of the target task to perform parameter tuning on the teacher language model to obtain an adjusted teacher language model 41. The adjusted teacher language model 41 is used to perform knowledge distillation on the incrementally pre-trained student language model 44. Before performing knowledge distillation on the incrementally pre-trained student language model using the teacher language model 41, the teacher language model is further fine-tuned using the third training sample set of the target task to improve the accuracy of the teacher language model 41, thereby further improving the accuracy of the distilled student language model obtained therefrom.

[0168] In some embodiments, the method further includes the following steps:

[0169] 1. Divide the training corpus for the target task into at least one training corpus segment;

[0170] 2. Map the training corpus fragments to obtain the corresponding word segmentation vector, semantic vector, and position vector; where the word segmentation vector is used to indicate the word segmentation features of the training corpus fragment, the semantic vector is used to indicate the semantic features of the training corpus fragment, and the position vector is used to indicate the position features of the training corpus fragment.

[0171] 3. Based on the word segmentation vectors, semantic vectors, and position vectors corresponding to the training corpus fragments, generate model input vectors corresponding to the training corpus fragments. The model input vectors are used to input the student language model and / or the teacher language model for model training.

[0172] In this implementation, the method for annotating training corpus fragments is a Chinese word segmentation information processing method based on BIOES (Bbegin, Inside, Outside, End, Single) encoding. In addition to mapping the semantic vector and position vector corresponding to the training corpus fragments, it also obtains the word segmentation vector corresponding to the training corpus fragments through mapping, thereby introducing the word segmentation features of the training corpus of the target task, and further improving the accuracy of the student language model and / or teacher language model obtained based on the training corpus of the target task.

[0173] In one example, such as Figure 5As shown, for the word "Shanghai," the teacher's language model and / or the student's language model will output annotation results [NR, NR] for "Shang" and "Hai" respectively. Then, a pooling operation is performed on [NR, NR], such as selecting the label information of the first character or the label information with the most occurrences as the label information of the entire word. This annotation method includes the following steps:

[0174] 1. Encode the word segmentation information of the training corpus fragments using BIOES.

[0175] The code for the beginning of a word is B, the code for the end of a word is E, the code for the middle part of a word is I, the code for single-character words is S, and the code for others is O.

[0176] 2. Map the BIOES sequence of the training corpus fragments to word segmentation vector 51, and input the vector obtained by adding the word segmentation vector 51, semantic vector 52 and position vector 53 of the training corpus fragments into the student language model and / or teacher language model.

[0177] 3. After obtaining the intermediate outputs of the student language model and / or the teacher language model, the annotation results corresponding to the beginning of each word in the training corpus segment are used as the annotation results of the entire word. Based on this, the corresponding loss is determined and the parameters of the student language model and / or the teacher language model are updated.

[0178] Below, in conjunction with Figure 6 The embodiments of this application will be summarized and introduced. For example... Figure 6 As shown, if incremental pre-training corpus exists, the language model training method includes the following steps (601-604):

[0179] Step 601: Obtain the fully trained student language model obtained after full pre-training.

[0180] Step 602: Use incremental pre-training corpus to train the student language model after full training to obtain the incremental pre-trained student language model.

[0181] Step 603: Based on the teacher model and knowledge distillation corpus 42, perform knowledge distillation on the incrementally pre-trained student language model to obtain the distilled student language model.

[0182] Step 604: Using the second training sample set of the target task, the parameters of the distilled student language model are adjusted to obtain the trained student language model.

[0183] In some embodiments, after step 601, if there is no incremental pre-training corpus, the language model training method further includes the following steps (605-606):

[0184] Step 605: Based on the teacher model and knowledge distillation corpus 42, perform knowledge distillation on the fully pre-trained student language model to obtain the distilled full student language model.

[0185] Step 606: Using the second training sample set of the target task, the parameters of the distilled full student language model are adjusted to obtain the student language model that has completed full training.

[0186] Under specific testing conditions, the student language model trained using the method provided in the embodiments of this application can achieve an inference speed 8 to 10 times that of the teacher language model, and a prediction accuracy of about 97% of that of the teacher model.

[0187] Based on actual testing, the student language model provided by this technical solution, after training, improves the prediction accuracy on the public test set by about 1% compared with student language models in related technologies.

[0188] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0189] Please refer to Figure 7 This diagram illustrates a block diagram of a language model training apparatus according to an embodiment of this application. The apparatus has the functionality to implement the language model training method example described above; this functionality can be implemented in hardware or by hardware executing corresponding software. The apparatus 700 can be the computer device described above, or it can be mounted on a computer device. The apparatus 700 may include: a full pre-training module 710, an incremental pre-training module 720, a model distillation module 730, and a parameter adjustment module 740.

[0190] The full pre-training module 710 is used to perform full pre-training on the initial student language model based on the full pre-training corpus to obtain the fully pre-trained student language model.

[0191] The incremental pre-training module 720 is used to perform incremental pre-training on the fully pre-trained student language model based on the incremental pre-training corpus to obtain the incrementally pre-trained student language model; wherein, the incremental pre-training corpus is a pre-training corpus added on the basis of the fully pre-training corpus.

[0192] The model distillation module 730 is used to perform knowledge distillation on the incrementally pre-trained student language model using the teacher language model to obtain the distilled student language model; wherein, the teacher language model is a language model trained using the first training sample set of the target task.

[0193] The parameter adjustment module 740 is used to perform parameter adjustment processing on the distilled student language model using the second training sample set of the target task to obtain a trained student language model.

[0194] In summary, the technical solution provided in this application embodiment, after obtaining the incremental pre-training corpus, uses only the incremental pre-training corpus to perform incremental pre-training on the student language model based on the fully pre-trained student language model. The original fully pre-trained corpus does not need to participate in the incremental pre-training process, saving incremental pre-training time and thus saving the overall training time of the student language model.

[0195] In an exemplary embodiment, such as Figure 8 As shown, the incremental pre-training module 720 includes: a result acquisition submodule 721, a loss determination submodule 722, and a parameter adjustment submodule 723.

[0196] The result acquisition submodule 721 is used to acquire a first prediction result and a second prediction result. The first prediction result refers to the prediction result of the student language model in incremental pre-training on the incremental pre-training corpus, and the second prediction result refers to the prediction result of the student language model after full pre-training on the incremental pre-training corpus.

[0197] The loss determination submodule 722 is used to determine a first loss based on the first prediction result, the second prediction result, and the label information of the incremental pre-training corpus; wherein the first loss is related to the degree of difference between the first prediction result and the label information of the incremental pre-training corpus, and is also related to the degree of difference between the first prediction result and the second prediction result.

[0198] The parameter adjustment submodule 723 is used to adjust the parameters of the student language model in the incremental pre-training based on the first loss until the first stopping condition is met, so as to obtain the incrementally pre-trained student language model.

[0199] In an exemplary embodiment, such as Figure 8 As shown, the loss determination submodule 722 includes: loss determination unit 7221.

[0200] The loss determination unit 7221 is used to determine a first sub-loss based on the first prediction result and the label information of the incremental pre-training corpus, wherein the first sub-loss is related to the degree of difference between the first prediction result and the label information of the incremental pre-training corpus.

[0201] The loss determination unit 7221 is further configured to determine a second sub-loss based on the first prediction result and the second prediction result, wherein the second sub-loss is related to the degree of difference between the first prediction result and the second prediction result.

[0202] The loss determination unit 7221 is further configured to determine the first loss based on the first sub-loss and the second sub-loss.

[0203] In an exemplary embodiment, such as Figure 8 As shown, the device 700 further includes a data acquisition module 750 and a loss determination module 760.

[0204] The data acquisition module 750 is used to acquire first hidden layer state data and second hidden layer state data. The first hidden layer state data refers to the hidden layer state data of the student language model in the incremental pre-training when generating the first prediction result, and the second hidden layer state data refers to the hidden layer state data of the student language model after full pre-training when generating the second prediction result.

[0205] The loss determination module 760 is used to determine a third sub-loss based on the first hidden layer state data and the second hidden layer state data, wherein the third sub-loss is related to the degree of difference between the first hidden layer state data and the second hidden layer state data.

[0206] The loss determination unit 7221 includes: a loss determination subunit 72211.

[0207] The loss determination subunit 72211 is used to determine the first loss based on the first sub-loss, the second sub-loss, and the third sub-loss.

[0208] In an exemplary embodiment, such as Figure 8 As shown, the loss determination subunit 72211 is used for:

[0209] Obtain the weight values ​​corresponding to the first sub-loss, the second sub-loss, and the third sub-loss respectively;

[0210] The first loss is obtained by weighting and summing the first sub-loss, the second sub-loss, and the third sub-loss based on the weight values.

[0211] In an exemplary embodiment, such as Figure 8 As shown, the result acquisition submodule 721 is used for:

[0212] The target text in the incremental pre-training corpus is masked to generate masked incremental pre-training corpus.

[0213] The masked incremental pre-training corpus is processed by the student language model in the incremental pre-training to obtain the first prediction result, which includes the masked text predicted by the student language model in the incremental pre-training.

[0214] The masked incremental pre-training corpus is processed by the fully pre-trained student language model to obtain the second prediction result, which includes the masked text predicted by the fully pre-trained student language model.

[0215] In an exemplary embodiment, such as Figure 8 As shown, the model distillation module 730 includes: a label determination submodule 731.

[0216] The label determination submodule 731 is used to determine first label information based on the teacher language model and the knowledge distillation corpus; wherein, the first label information refers to the label information inferred by the teacher language model on the knowledge distillation corpus.

[0217] The label determination submodule 731 is further configured to determine multiple second label information based on multiple incrementally pre-trained student language models and the knowledge distillation corpus; wherein, the second label information refers to the label information inferred by the incrementally pre-trained student language models on the knowledge distillation corpus.

[0218] The loss determination submodule 722 is further configured to, for the i-th incrementally pre-trained student language model, determine the second loss corresponding to the i-th incrementally pre-trained student language model based on the first label information, the second label information corresponding to the i-th incrementally pre-trained student language model, and the second label information corresponding to other incrementally pre-trained student language models; wherein, the second loss corresponding to the i-th incrementally pre-trained student language model is related to the degree of difference between the first label information and the second label information corresponding to the i-th incrementally pre-trained student language model, and is also related to the degree of difference between the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to other incrementally pre-trained student language models, and i is a positive integer.

[0219] The parameter adjustment submodule 723 is further configured to adjust the parameters of the student language model after the i-th incremental pre-training based on the second loss corresponding to the i-th incremental pre-trained student language model, until the second stopping condition is met, so as to obtain the i-th distilled student language model.

[0220] In an exemplary embodiment, such as Figure 8As shown, the loss determination unit 7221 is further configured to determine a fourth sub-loss based on the first label information and the second label information corresponding to the i-th incrementally pre-trained student language model, wherein the fourth sub-loss is related to the degree of difference between the first label information and the second label information corresponding to the i-th incrementally pre-trained student language model.

[0221] The loss determination unit 7221 is further configured to determine a fifth sub-loss based on the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to the other incrementally pre-trained student language models. The fifth sub-loss is related to the degree of difference between the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to the other incrementally pre-trained student language models.

[0222] The loss determination unit 7221 is further configured to determine the second loss corresponding to the i-th incremental pre-trained student language model based on the fourth sub-loss and the fifth sub-loss.

[0223] In an exemplary embodiment, such as Figure 8 As shown, the loss determination unit 7221 is used for:

[0224] Determine the KL divergence between the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to the other incrementally pre-trained student language models;

[0225] The fifth sub-loss is determined based on the KL divergence.

[0226] In an exemplary embodiment, the label information is soft label information, which refers to the feature map generated by the model during the reasoning process on the knowledge distillation corpus.

[0227] In an exemplary embodiment, such as Figure 8 As shown, the parameter adjustment module 740 is further configured to use the third training sample set of the target task to perform parameter adjustment processing on the teacher language model to obtain the adjusted teacher language model; wherein, the adjusted teacher language model is used to perform knowledge distillation processing on the incrementally pre-trained student language model.

[0228] In an exemplary embodiment, such as Figure 8 As shown, the parameter adjustment module 740 is used for:

[0229] The training corpus for the target task is divided into at least one training corpus segment;

[0230] The training corpus fragments are mapped to obtain the word segmentation vector, semantic vector, and position vector corresponding to the training corpus fragments; wherein, the word segmentation vector is used to indicate the word segmentation features of the training corpus fragments, the semantic vector is used to indicate the semantic features of the training corpus fragments, and the position vector is used to indicate the position features of the training corpus fragments.

[0231] Based on the word segmentation vector, semantic vector, and position vector corresponding to the training corpus fragment, a model input vector corresponding to the training corpus fragment is generated. The model input vector is used to input the student language model and / or the teacher language model for model training.

[0232] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0233] Please refer to Figure 9 This diagram illustrates a structural block diagram of a computer device according to an embodiment of this application. The computer device is used to implement the language model training method provided in the above embodiments. Specifically:

[0234] The computer device 900 includes a CPU (Central Processing Unit) 901, a system memory 904 including RAM (Random Access Memory) 902 and ROM (Read-Only Memory) 903, and a system bus 905 connecting the system memory 904 and the central processing unit 901. The computer device 900 also includes a basic I / O (Input / Output) system 906 that facilitates information transfer between various components within the computer, and a mass storage device 907 for storing the operating system 913, application programs 914, and other program modules 915.

[0235] The basic input / output system 906 includes a display 908 for displaying information and an input device 909 for user input, such as a mouse or keyboard. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include the input / output controller 910 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, printer, or other types of output devices.

[0236] The mass storage device 907 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. That is, the mass storage device 907 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0237] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 904 and mass storage device 907 described above can be collectively referred to as memory.

[0238] According to various embodiments of this application, the computer device 900 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 900 can be connected to a network 912 via a network interface unit 911 connected to the system bus 905, or the network interface unit 911 can be used to connect to other types of networks or remote computer systems (not shown).

[0239] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is executed by a processor to implement the language model training method described above.

[0240] In an exemplary embodiment, a computer program product is also provided, which, when executed by a processor, is used to implement the language model training method described above.

[0241] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Other embodiments of this application will readily conceive of by those skilled in the art upon considering the specification and practicing the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0242] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A language model training method, characterized in that, The method includes: The initial student language model is fully pre-trained based on the full pre-training corpus to obtain the fully pre-trained student language model. Obtain a first prediction result and a second prediction result. The first prediction result refers to the prediction result of the student language model in incremental pre-training on the incremental pre-training corpus, and the second prediction result refers to the prediction result of the student language model after full pre-training on the incremental pre-training corpus. The incremental pre-training corpus is a pre-training corpus added on the basis of the full pre-training corpus, and the initial model of the student language model in incremental pre-training is the student language model after full pre-training. Based on the first prediction result, the second prediction result, and the label information of the incremental pre-training corpus, a first loss is determined; wherein, the first loss is related to the degree of difference between the first prediction result and the label information of the incremental pre-training corpus, and is also related to the degree of difference between the first prediction result and the second prediction result; The parameters of the student language model in the incremental pre-training are adjusted based on the first loss until the first stopping condition is met, thus obtaining the incrementally pre-trained student language model. The incrementally pre-trained student language model is subjected to knowledge distillation using the teacher language model to obtain the distilled student language model; wherein, the teacher language model is a language model trained using the first training sample set of the target task; Using the second training sample set of the target task, the parameters of the distilled student language model are adjusted to obtain a fully trained student language model.

2. The method according to claim 1, characterized in that, The determination of the first loss based on the first prediction result, the second prediction result, and the label information of the incremental pre-training corpus includes: Based on the first prediction result and the label information of the incremental pre-training corpus, a first sub-loss is determined, wherein the first sub-loss is related to the degree of difference between the first prediction result and the label information of the incremental pre-training corpus; Based on the first prediction result and the second prediction result, a second sub-loss is determined, which is related to the degree of difference between the first prediction result and the second prediction result. The first loss is determined based on the first sub-loss and the second sub-loss.

3. The method according to claim 2, characterized in that, The method further includes: Obtain first hidden state data and second hidden state data. The first hidden state data refers to the hidden state data of the student language model in the incremental pre-training when generating the first prediction result, and the second hidden state data refers to the hidden state data of the student language model after full pre-training when generating the second prediction result. Based on the first hidden layer state data and the second hidden layer state data, a third sub-loss is determined, which is related to the degree of difference between the first hidden layer state data and the second hidden layer state data; Determining the first loss based on the first sub-loss and the second sub-loss includes: The first loss is determined based on the first sub-loss, the second sub-loss, and the third sub-loss.

4. The method according to claim 3, characterized in that, Determining the first loss based on the first sub-loss, the second sub-loss, and the third sub-loss includes: Obtain the weight values ​​corresponding to the first sub-loss, the second sub-loss, and the third sub-loss respectively; The first loss is obtained by weighting and summing the first sub-loss, the second sub-loss, and the third sub-loss based on the weight values.

5. The method according to claim 1, characterized in that, The process of obtaining the first prediction result and the second prediction result includes: The target text in the incremental pre-training corpus is masked to generate masked incremental pre-training corpus. The masked incremental pre-training corpus is processed by the student language model in the incremental pre-training to obtain the first prediction result, which includes the masked text predicted by the student language model in the incremental pre-training. The masked incremental pre-training corpus is processed by the fully pre-trained student language model to obtain the second prediction result, which includes the masked text predicted by the fully pre-trained student language model.

6. The method according to claim 1, characterized in that, The step of using the teacher's language model to perform knowledge distillation on the incrementally pre-trained student language model to obtain the distilled student language model includes: Based on the teacher language model and the knowledge distillation corpus, a first label information is determined; wherein, the first label information refers to the label information inferred by the teacher language model on the knowledge distillation corpus; Based on multiple incrementally pre-trained student language models and the knowledge distillation corpus, multiple second label information is determined; wherein, the second label information refers to the label information inferred by the incrementally pre-trained student language models on the knowledge distillation corpus; For the i-th incrementally pre-trained student language model, based on the first label information, the second label information corresponding to the i-th incrementally pre-trained student language model, and the second label information corresponding to other incrementally pre-trained student language models, a second loss corresponding to the i-th incrementally pre-trained student language model is determined; wherein, the second loss corresponding to the i-th incrementally pre-trained student language model is related to the degree of difference between the first label information and the second label information corresponding to the i-th incrementally pre-trained student language model, and is also related to the degree of difference between the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to other incrementally pre-trained student language models, and i is a positive integer; Based on the second loss corresponding to the i-th incremental pre-trained student language model, the parameters of the i-th incremental pre-trained student language model are adjusted until the second stopping condition is met, thus obtaining the i-th distilled student language model.

7. The method according to claim 6, characterized in that, The step of determining the second loss corresponding to the i-th incrementally pre-trained student language model based on the first label information, the second label information corresponding to the i-th incrementally pre-trained student language model, and the second label information corresponding to other incrementally pre-trained student language models includes: Based on the first label information and the second label information corresponding to the i-th incrementally pre-trained student language model, a fourth sub-loss is determined, wherein the fourth sub-loss is related to the degree of difference between the first label information and the second label information corresponding to the i-th incrementally pre-trained student language model; Based on the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to the other incrementally pre-trained student language models, a fifth sub-loss is determined. The fifth sub-loss is related to the degree of difference between the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to the other incrementally pre-trained student language models. Based on the fourth sub-loss and the fifth sub-loss, the second loss corresponding to the student language model after the i-th incremental pre-training is determined.

8. The method according to claim 7, characterized in that, The determination of the fifth sub-loss based on the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to the other incrementally pre-trained student language models includes: Determine the KL divergence between the second label information corresponding to the i-th incrementally pre-trained student language model and the second label information corresponding to the other incrementally pre-trained student language models; The fifth sub-loss is determined based on the KL divergence.

9. The method according to claim 6, characterized in that, The label information is soft label information, which refers to the feature map generated by the model during the reasoning process on the knowledge distillation corpus.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: Using the third training sample set of the target task, the teacher language model is subjected to parameter adjustment processing to obtain the adjusted teacher language model; The adjusted teacher language model is used to perform knowledge distillation on the incrementally pre-trained student language model.

11. The method according to any one of claims 1 to 9, characterized in that, The method further includes: The training corpus for the target task is divided into at least one training corpus segment; The training corpus fragments are mapped to obtain the word segmentation vector, semantic vector, and position vector corresponding to the training corpus fragments; wherein, the word segmentation vector is used to indicate the word segmentation features of the training corpus fragments, the semantic vector is used to indicate the semantic features of the training corpus fragments, and the position vector is used to indicate the position features of the training corpus fragments. Based on the word segmentation vector, semantic vector, and position vector corresponding to the training corpus fragment, a model input vector corresponding to the training corpus fragment is generated. The model input vector is used to input the student language model and / or the teacher language model for model training.

12. A language model training device, characterized in that, The device includes: The full pre-training module is used to perform full pre-training on the initial student language model based on the full pre-training corpus, so as to obtain the fully pre-trained student language model. An incremental pre-training module is used to obtain a first prediction result and a second prediction result. The first prediction result refers to the prediction result of the student language model in incremental pre-training on the incremental pre-training corpus, and the second prediction result refers to the prediction result of the student language model after full pre-training on the incremental pre-training corpus. The incremental pre-training corpus is pre-training corpus added on top of the full pre-training corpus, and the initial model of the student language model in incremental pre-training is the student language model after full pre-training. A first loss is determined based on the first prediction result, the second prediction result, and the label information of the incremental pre-training corpus. The first loss is related to the degree of difference between the first prediction result and the label information of the incremental pre-training corpus, and also to the degree of difference between the first prediction result and the second prediction result. The parameters of the student language model in incremental pre-training are adjusted based on the first loss until a first stopping condition is met, thus obtaining the incrementally pre-trained student language model. The model distillation module is used to perform knowledge distillation on the incrementally pre-trained student language model using the teacher language model to obtain the distilled student language model; wherein, the teacher language model is a language model trained using the first training sample set of the target task; The parameter adjustment module is used to adjust the parameters of the distilled student language model using the second training sample set of the target task, so as to obtain a trained student language model.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the language model training method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the language model training method as described in any one of claims 1 to 11.

Images