Language model training method, NLP task processing method and device

Abstract

The present application provides a method for training a language model, a method for processing an NLP task, and a device, including: obtaining a training sample set; the training sample set includes a first task label of a new task, multiple first training texts of the new task and a first text label of each first training text, and a second task label of each old task in at least one old task; copying the language model to obtain a teacher language model, and using the language model as a student language model; inputting the second task label into the teacher language model, generating multiple second training texts corresponding to the old task and a second text label of each second training text; inputting the first task label, the second task label, the first training text, and the second training text into the student language model, generating a first predicted text, a first predicted result, a second predicted text, and a second predicted result, and training the student language model. According to the embodiments of the present application, the problem of large storage resource usage in the related art can be solved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a language model training method, an NLP task processing method and a device. Background Art

[0002] With the development of natural language processing (NLP) technology, language models have been widely used in various fields of life to complete different types of natural language processing tasks, such as text classification, sentiment classification, semantic role labeling of dialogue systems, and so on.

[0003] At present, after learning a new task, the performance of a conventional neural network on an old task will drop sharply, that is, the performance of the neural network will drop as the distribution of the learning data set changes. Therefore, the neural network has the problem of catastrophic forgetting. In related technologies, in order to avoid catastrophic forgetting of neural networks, multiple sub-neural network models can be added to the main neural network model, and different sub-neural network models learn different tasks. It is also possible to review the tasks that have been trained by saving the training samples and prediction results of the tasks that have been trained.

[0004] However, in the above implementation method of avoiding catastrophic forgetting of the neural network, the model increases as the sub-networks are added, and the storage resources occupied by the neural network also increase; saving the training samples and prediction results of the tasks that have been trained requires a large amount of storage resources. Summary of the invention

[0005] The embodiments of the present application provide a language model training method, an NLP task processing method and a device, which can generate training texts for trained tasks through a teacher language model, thereby solving the problem of large storage resource usage in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a method for training a language model, the method comprising:

[0007] Obtain a training sample set; the training sample set includes a first task label of a new task, a plurality of first training texts of the new task and a first text label of each first training text, and a second task label of each old task in at least one old task, the old task is an NLP task for which a language model has been trained, and the new task is an NLP task for which a language model has not been trained;

[0008] Copy the language model to obtain the teacher language model, and use the language model as the student language model;

[0009] Inputting the second task label into the teacher language model to generate a plurality of second training texts corresponding to the old task and a second text label for each second training text;

[0010] Inputting the first task label, the second task label, the first training text, and the second training text into the student language model, generating a first predicted text corresponding to the first task label, a first predicted result corresponding to the first training text, a second predicted text corresponding to the second task label, and a second predicted result corresponding to the second training text;

[0011] The student language model is trained according to the first training text and the first predicted text, the second training text and the second predicted text, the first prediction result and the first text label, and the second prediction result and the second text label.

[0012] In a possible implementation, the first training text and the second training text are both texts in question-answer format, and the question-answer format texts include question prompt information and questions generated according to the question prompt information.

[0013] In a possible implementation, training the student language model according to the first training text, the first prediction text, the second training text, and the second prediction text includes:

[0014] Perform knowledge distillation on the student language model according to the second text label and the second prediction result;

[0015] The student language model is trained based on the loss between the first training text and the first predicted text, the loss between the second training text and the second predicted text, the loss between the first prediction result and the first text label, and the loss between the second prediction result and the second text label.

[0016] In a possible implementation, knowledge distillation is performed on the student language model according to the second text label and the second prediction result, including:

[0017] Calculate the JS divergence between the teacher language model and the student language model according to the second text label and the second prediction result;

[0018] Calculate the earth shift distance of feature transfer from each layer in the teacher language model to each layer in the student language model;

[0019] Calculate the loss between the teacher language model and the student language model based on JS divergence and earth mover distance;

[0020] Update the student language model based on the loss between the teacher language model and the student language model.

[0021] In a possible implementation, the second task label is input into the teacher language model to generate a plurality of second predicted texts corresponding to the new task, including:

[0022] The second task label is input into the teacher language model, and a plurality of second predicted texts are generated according to the number and a preset ratio of the second training texts in the second training text set.

[0023] In a second aspect, an embodiment of the present application provides a method for processing an NLP task, including:

[0024] Get the text for NLP tasks;

[0025] The text is input into the student language model in the first aspect or any possible implementation of the first aspect to generate a prediction result of the NLP task.

[0026] In a possible implementation, the text includes question prompt information and questions of the NLP task; the text is input into the student language model in the first aspect or any possible implementation of the first aspect to generate a prediction result of the NLP task, including:

[0027] Input the question prompt information and the question into the student language model to generate the answer corresponding to the question.

[0028] In a third aspect, an embodiment of the present application provides a language model training device, including:

[0029] An acquisition module is used to acquire a training sample set; the training sample set includes a first task label of a new task, a plurality of first training texts of the new task and a first text label of each first training text, and a second task label of each old task in at least one old task, the old task is an NLP task for which a language model has been trained, and the new task is an NLP task for which a language model has not been trained;

[0030] A copy module is used to copy the language model to obtain the teacher language model, and use the language model as the student language model;

[0031] A generating module, used for inputting the second task label into the teacher language model, generating a plurality of second training texts corresponding to the old task and a second text label for each second training text;

[0032] The generation module is also used to input the first task label, the second task label, the first training text and the second training text into the student language model to generate a first predicted text corresponding to the first task label, a first predicted result corresponding to the first training text, a second predicted text corresponding to the second task label and a second predicted result corresponding to the second training text;

[0033] The training module is used to train the student language model according to the first training text and the first predicted text, the second training text and the second predicted text, the first prediction result and the first text label, and the second prediction result and the second text label.

[0034] In a possible implementation, the first training text and the second training text are both texts in question-answer format, and the question-answer format texts include question prompt information and questions generated according to the question prompt information.

[0035] In a possible implementation, the training module is used to:

[0036] Perform knowledge distillation on the student language model according to the second text label and the second prediction result;

[0037] The student language model is trained based on the loss between the first training text and the first predicted text, the loss between the second training text and the second predicted text, the loss between the first prediction result and the first text label, and the loss between the second prediction result and the second text label.

[0038] In a possible implementation, the training module is used to:

[0039] Calculate the JS divergence between the teacher language model and the student language model according to the second text label and the second prediction result;

[0040] Calculate the earth shift distance of feature transfer from each layer in the teacher language model to each layer in the student language model;

[0041] Calculate the loss between the teacher language model and the student language model based on JS divergence and earth mover distance;

[0042] Update the student language model based on the loss between the teacher language model and the student language model.

[0043] In a possible implementation, the generation module is used to input the second task label into the teacher language model, and generate a plurality of second predicted texts according to the number and a preset ratio of the second training texts in the second training text set.

[0044] In a fourth aspect, an embodiment of the present application provides an NLP task processing device, including:

[0045] Acquisition module, used to obtain text for NLP tasks;

[0046] A generation module is used to input text into a student language model as described in any one of claims 1-5 to generate a prediction result of an NLP task.

[0047] In a possible implementation, the text includes question prompt information and questions of the NLP task; the generation module is used to: input the question prompt information and questions into the student language model to generate answers corresponding to the questions.

[0048] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the computer program implements the method provided in the first aspect or any possible implementation of the first aspect, or implements the method provided in the second aspect or any possible implementation of the second aspect.

[0049] In a sixth aspect, an embodiment of the present application provides a computer storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the method provided in the above-mentioned first aspect or any possible implementation of the first aspect, or implements the method provided in the above-mentioned second aspect or any possible implementation of the second aspect.

[0050] The training method of the language model, the NLP task processing method and the device provided in the embodiment of the present application, by inputting the second task label of the old task into the teacher language model trained by the old task, thereby generating the second training text corresponding to the old task and the text label corresponding to the second training text. Among them, the new task is an NLP task that the language model has not been trained, and the old task is an NLP task that the language model has been trained. The teacher language model is obtained by copying the language model, and the student language model is the language model. Then the first training text and the first task label, the second training text and the second task label of the new task are input into the student language model to generate the first predicted text corresponding to the first task label and the second predicted text corresponding to the second task label. The student language model is trained according to the first training text, the first predicted text, the second training text, the second predicted text, the first text label and the second text label. In this way, there is no need to store the training texts corresponding to the tasks that have been trained by the teacher language model. Instead, the training texts corresponding to the trained tasks can be obtained, and the student language model that is the same as the teacher language model can be trained in combination with the training texts corresponding to the tasks that have not been trained by the model. This will result in a model that can perform both trained and untrained tasks, avoiding the problem of catastrophic forgetting of the model and improving the performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a flow chart of a method for training a language model provided in an embodiment of the present application is shown;

[0052] Figure 2A schematic diagram of a process flow of an NLP task processing method provided in an embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of the structure of a language model training device provided in an embodiment of the present application is shown;

[0054] Figure 4 A schematic diagram of the structure of an NLP task processing device provided in an embodiment of the present application is shown;

[0055] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0057] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0058] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple screen terminals refers to two or more screen terminals.

[0059] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0060] With the development of natural language processing (NLP) technology, language models have been widely used in various fields of life to complete different types of natural language processing tasks, such as text classification, sentiment classification, semantic role labeling of dialogue systems, and so on.

[0061] At present, due to the problem of catastrophic forgetting in neural networks, the performance of neural networks will be reduced. In related technologies, in order to avoid catastrophic forgetting in neural networks, multiple sub-neural network models can be added to the main neural network model, and different sub-neural network models learn different tasks. It is also possible to review the tasks that have been trained by saving the training samples and prediction results of the tasks that have been trained.

[0062] However, in the above implementation method of avoiding catastrophic forgetting of the neural network, the model increases as the sub-networks are added, and the storage resources occupied by the neural network also increase; saving the training samples and prediction results of the tasks that have been trained requires a large amount of storage resources.

[0063] Based on this, the embodiments of the present application provide a language model training method, an NLP task processing method and a device, which generate training texts for trained tasks through a teacher language model, thereby avoiding the problem of large storage resources occupied by storing training sample data.

[0064] Figure 1 is a flow chart of a method for training a language model provided in an embodiment of the present application. Figure 1 As shown, the language model training method provided in the embodiment of the present application may include S101-S104.

[0065] S101: Obtain a training sample set; the training sample set includes a first task label of a new task, multiple first training texts of the new task and a first text label of each first training text, and a second task label of each old task in at least one old task, the old task is an NLP task for which a language model has been trained, and the new task is an NLP task for which a language model has not been trained.

[0066] NLP tasks can involve many aspects, such as mathematics, poetry, music, etc. In order to distinguish between different types of NLP, a task label can be set for each NLP task, and the task label can characterize the task type. Here, the old task is an NLP task that has been trained by a language model, and the new task is an NLP task that has not been trained by a language model. For example, the new task is a task in mathematics, and the old task is a task in poetry. The first training text of the new task can be "1+1 equals 2".

[0067] Here, there can be multiple old tasks.

[0068] In some embodiments, in order to improve the performance of the model, after the first training text is obtained, the training text can be formatted. Specifically, a regularization method can be used to unify the first training text according to the question-answer format into a question-answer format text. Here, the question-answer format text includes question prompt information and questions generated according to the question prompt information. For example, if the text of the new task is "addition, 1+1 equals 2", the text of the new task can be unified into "C#1+1, Q# equals, A#2", then the first training text of the new task is "C#1+1, Q# equals", and the first text label of the first training text is "A#2".

[0069] Here, the first training text may include a first text label. For example, if the text of the new task is "addition, 1+1 equals 2", the text of the new task can be unified as "C#1+1, Q# equals a few, A#2", and the first training text can be determined as "C#1+1, Q# equals a few, A#2".

[0070] In some embodiments, the language model is a GPT-2 model.

[0071] S102: Copy the language model to obtain a teacher language model, and use the language model as a student language model.

[0072] In order to achieve self-distillation of the language model and avoid the language model being unable to perform old tasks due to catastrophic forgetting, it is also possible to copy the language model, use the copied language model as the teacher language model, and then use the language model itself as the student language model.

[0073] Here, in order to avoid occupying storage resources, the teacher language model can be deleted after the student language model is trained.

[0074] S103: Input the second task label into the teacher language model to generate a plurality of second training texts corresponding to the old task and a second text label for each second training text.

[0075] In order to avoid catastrophic forgetting of the student language model during the training of the new task and loss of performance of the old task, the training text of the new task and the training text of the old task can be combined during the training process. The second task label of the old task is input into the teacher language model to obtain pseudo data corresponding to the old task. This pseudo data can be used as the training sample of the old task for training the student language model, that is, the second training text and the second text label. The second training text is the predicted text of the second task label of the old task randomly generated by the teacher language model according to the second task label.

[0076] As the number of different tasks increases, the number of training samples will also increase, and the occupied storage space will become larger and larger. The embodiment of the present application generates pseudo data by a teacher language model that has trained the old tasks, and obtains the training text of the old tasks for training the student language model. In this way, there is no need to allocate storage space to store a large number of training samples of the old tasks, thus avoiding the problem of large storage resource occupation.

[0077] In some embodiments, the text format of the second training text is a question-answer format. For example, if the second task tag is mathematics, the generated second training text may be "1+1=2", "1*1=1".

[0078] In order to ensure the performance of the model, the second training text can be formatted in a unified format. For example, "1+1=2" can be unified into "C#1+1, Q# equals to" and "A#2", and "1*1=1" can be unified into "C#1*1, Q# equals to" and "A#1", etc. The second text label represents the true value corresponding to the question. For example, "X" in "A#X" represents the true value corresponding to the second training text.

[0079] In some embodiments, in order to improve the training efficiency of the model, the second training text may be generated according to the number of the first training texts. Specifically, the product of the number of the first training texts and the preset ratio is calculated to determine the number of the second training texts required.

[0080] S104: Input the first task label, the second task label, the first training text and the second training text into the student language model to generate a first predicted text corresponding to the first task label, a first predicted result corresponding to the first training text, a second predicted text corresponding to the second task label and a second predicted result corresponding to the second training text.

[0081] Here, the second predicted text refers to the predicted text corresponding to the second task label generated by the student language model based on the second task label.

[0082] Specifically, the first task label is input into the student language model to generate the first predicted text; the second task label is input into the student language model to generate the second predicted text, that is, the student language model performs the language task. The first training text is input into the student language model, and the first predicted text corresponding to the first training text can be obtained. For example, if the first training text is "C#Quiet Night Thoughts, Q#Who is the author", the first predicted text can be "A#Du Fu". The second training text is input into the student language model, and the second predicted text corresponding to the second training text can be obtained. That is, the student language model performs the question-answering task.

[0083] S105: Training the student language model according to the first training text and the first predicted text, the second training text and the second predicted text, the first prediction result and the first text label, and the second prediction result and the second text label.

[0084] In some embodiments, in order to ensure that the student language model can retain the performance of the teacher language model for the old task, it is necessary to perform knowledge distillation on the student language model so that the language model completes self-distillation. Specifically, in S105, first, knowledge distillation is performed on the student language model according to the second predicted text and the second text label; then, the language model is trained according to the loss between the first predicted text and the first training text, the loss between the first text label and the first predicted result, the loss between the second predicted text and the second training text, and the loss between the second text label and the second predicted result.

[0085] Here, in the process of knowledge distillation of the student language model, first, the JS (Jensen-Shannon) divergence between the teacher language model and the student language model is calculated based on the second prediction text and the second training text. And the earth moving distance of the feature transfer of each layer in the teacher language model to each layer in the student language model is calculated; then, according to the JS divergence and the earth moving distance, the loss between the teacher language model and the student language model is calculated; according to the loss between the teacher language model and the student language model, the student language model is updated.

[0086] In this way, since the teacher language model is obtained by copying the language model, knowledge distillation is performed on the student model to achieve self-distillation of the language model, that is, the language model undergoes incremental learning, which improves the versatility of the language model in performing tasks.

[0087] The training method of the language model provided in the embodiment of the present application is to input the second task label of the old task into the teacher language model trained by the old task, thereby generating the second training text corresponding to the old task and the text label corresponding to the second training text. Wherein, the new task is an NLP task that the language model has not been trained, and the old task is an NLP task that the language model has been trained. The teacher language model is obtained by copying the language model, and the student language model is the language model. Then the first training text and the first task label, the second training text and the second task label of the new task are input into the student language model to generate the first predicted text corresponding to the first task label and the second predicted text corresponding to the second task label. The student language model is trained according to the first training text, the first predicted text, the second training text, the first text label and the second text label of the second predicted text. In this way, there is no need to store the training text corresponding to the task that the teacher language model has trained, and the training text corresponding to the task that has been trained can be obtained, and the student language model identical to the teacher language model is trained in combination with the training text corresponding to the task that the model has not trained, so as to obtain a model that can perform both the trained tasks and the untrained tasks, thereby avoiding the problem of catastrophic forgetting of the model and improving the performance of the model.

[0088] Based on the language model in the above embodiment, the embodiment of the present application also provides an NLP task processing method. Figure 2 is a flowchart of an NLP task processing method provided in an embodiment of the present application, such as Figure 2 As shown, the NLP task processing method provided in the embodiment of the present application may include S201 to S202.

[0089] S201: Obtain text for an NLP task.

[0090] In some embodiments, the NLP task text is a text in a question-and-answer format, and the text of the NLP task includes question prompt information and questions.

[0091] S202: Input the text into the language model to generate a prediction result of the NLP task.

[0092] Here, the language model is Figure 1 The language model after training of the corresponding embodiment.

[0093] In some embodiments, question prompt information and questions are input into a language model, and an answer corresponding to the question is generated through greedy decoding.

[0094] The NLP task processing method provided in the embodiment of the present application is to input the NLP task into the Figure 1 The language model trained in the corresponding embodiment obtains prediction results, thereby improving the extensiveness of task processing.

[0095] Based on the language model training method in the above embodiment, the embodiment of the present application also provides a language model training device. Figure 3 is a structural diagram of a language model training device 300 provided in an embodiment of the present application, such as Figure 3 As shown, the language model training device 300 may include an acquisition module 301 , a copy module 302 , a generation module 303 , and a training module 304 .

[0096] The acquisition module 301 is used to acquire a training sample set; the training sample set includes a first task label of a new task, multiple first training texts of the new task and a first text label of each first training text, and a second task label of each old task in at least one old task, the old task is an NLP task for which a language model has been trained, and the new task is an NLP task for which a language model has not been trained;

[0097] A copy module 302 is used to copy the language model to obtain a teacher language model, and use the language model as a student language model;

[0098] A generating module 303, used for inputting the second task label into the teacher language model, generating a plurality of second training texts corresponding to the old task and a second text label for each second training text;

[0099] The generation module 303 is further used to input the first task label, the second task label, the first training text and the second training text into the student language model, and generate a first predicted text corresponding to the first task label, a first predicted result corresponding to the first training text, a second predicted text corresponding to the second task label and a second predicted result corresponding to the second training text;

[0100] The training module 304 is used to train the student language model according to the first training text and the first predicted text, the second training text and the second predicted text, the first prediction result and the first text label, and the second prediction result and the second text label.

[0101] In a possible implementation, the first training text and the second training text are both texts in question-answer format, and the question-answer format texts include question prompt information and questions generated according to the question prompt information.

[0102] In a possible implementation, the training module 304 is used to:

[0103] Perform knowledge distillation on the student language model according to the second text label and the second prediction result;

[0104] The student language model is trained based on the loss between the first training text and the first predicted text, the loss between the second training text and the second predicted text, the loss between the first prediction result and the first text label, and the loss between the second prediction result and the second text label.

[0105] In a possible implementation, the training module 304 is used to:

[0106] Calculate the JS divergence between the teacher language model and the student language model according to the second text label and the second prediction result;

[0107] Calculate the earth shift distance of feature transfer from each layer in the teacher language model to each layer in the student language model;

[0108] Calculate the loss between the teacher language model and the student language model based on JS divergence and earth mover distance;

[0109] Update the student language model based on the loss between the teacher language model and the student language model.

[0110] In a possible implementation, the generating module 303 is used to input the second task label into the teacher language model, and generate a plurality of second predicted texts according to the number and a preset ratio of the second training texts in the second training text set.

[0111] The language model training device provided in the embodiment of the present application can perform Figure 1 The steps of the method in the corresponding embodiments can achieve the same technical effect, and to avoid repetition, they will not be described in detail here.

[0112] The training device of the language model provided in the embodiment of the present application generates the second training text corresponding to the old task and the text label corresponding to the second training text by inputting the second task label of the old task into the teacher language model trained by the old task. Wherein, the new task is an NLP task that the language model has not been trained, and the old task is an NLP task that the language model has been trained. The teacher language model is obtained by copying the language model, and the student language model is the language model. Then the first training text and the first task label, the second training text and the second task label of the new task are input into the student language model to generate the first predicted text corresponding to the first task label and the second predicted text corresponding to the second task label. The student language model is trained according to the first training text, the first predicted text, the second training text, the first text label and the second text label of the second predicted text. In this way, there is no need to store the training text corresponding to the task that the teacher language model has trained, and the training text corresponding to the task that has been trained can be obtained, and the student language model identical to the teacher language model is trained in combination with the training text corresponding to the task that the model has not trained, so as to obtain a model that can perform both the trained tasks and the untrained tasks, thereby avoiding the problem of catastrophic forgetting of the model and improving the performance of the model.

[0113] Based on the NLP task processing method in the above embodiment, the embodiment of the present application also provides an NLP task processing device. Figure 4 is a schematic diagram of the structure of the NLP task processing device 400 provided in an embodiment of the present application, such as Figure 4 As shown, the NLP task processing device 400 provided in the embodiment of the present application may include an acquisition module 401 and a generation module 402.

[0114] An acquisition module 401 is used to acquire text of an NLP task;

[0115] The generation module 402 is used to input the text into the student language model as described in any one of claims 1-5 to generate the prediction result of the NLP task.

[0116] In a possible implementation, the text includes question prompt information and questions of the NLP task; the generation module 402 is used to: input the question prompt information and questions into the student language model to generate answers corresponding to the questions.

[0117] The language model training device provided in the embodiment of the present application can perform Figure 2 The steps of the method in the corresponding embodiments can achieve the same technical effect, and to avoid repetition, they will not be described in detail here.

[0118] The NLP task processing device provided in the embodiment of the present application inputs the NLP task into the Figure 1The language model trained in the corresponding embodiment obtains prediction results, thereby improving the extensiveness of task processing.

[0119] A computer device provided in an embodiment of the present application is introduced below.

[0120] Figure 5 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 5 As shown, the computer device provided in the embodiment of the present application can be used to implement the language model training method or NLP task processing method described in the above method embodiment.

[0121] The computer device may include a processor 501 and a memory 502 storing computer program instructions.

[0122] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0123] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0124] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods in the present application.

[0125] The processor 501 implements any one of the language model training methods or NLP task processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 502.

[0126] In one example, the electronic device may further include a communication interface 505 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 505 are connected via a bus 510 and communicate with each other.

[0127] The communication interface 505 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0128] Bus 510 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0129] In addition, in combination with the above embodiments, the embodiments of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any language model training method or NLP task processing method in the above embodiments is implemented.

[0130] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0131] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0132] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.

[0133] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A method for training a language model, characterized in that: include: Acquire a training sample set; the training sample set includes a first task label of a new task, multiple first training texts of the new task and a first text label of each of the first training texts, and a second task label of each of at least one old task, wherein the old task is an NLP task for which a language model has been trained, and the new task is an NLP task for which the language model has not been trained, the first task label is used to indicate the task type to which the new task belongs, the second task label is used to indicate the task type to which the old task belongs, the new task and the old task have different task types, and the first text label is used to indicate a true value corresponding to the first training text; Copying the language model to obtain a teacher language model, and using the language model as a student language model; Inputting the second task label into the teacher language model to generate a plurality of second training texts corresponding to the old task and a second text label for each of the second training texts, wherein the second text label is used to indicate a true value corresponding to the second training text; Inputting the first task label, the second task label, the first training text, and the second training text into the student language model, generating a first predicted text corresponding to the first task label, a first predicted result corresponding to the first training text, a second predicted text corresponding to the second task label, and a second predicted result corresponding to the second training text; Training the student language model according to the first training text and the first predicted text, the second training text and the second predicted text, the first prediction result and the first text label, the second prediction result and the second text label; The first training text and the first text label constitute a question-answer format text, and the question-answer format text includes question prompt information and questions generated according to the question prompt information.

2. The method according to claim 1, characterized in that The training of the student language model according to the first training text, the first predicted text, the second training text and the second predicted text comprises: Performing knowledge distillation on the student language model according to the second text label and the second prediction result; The student language model is trained according to the loss between the first training text and the first predicted text, the loss between the second training text and the second predicted text, the loss between the first prediction result and the first text label, and the loss between the second prediction result and the second text label.

3. The method according to claim 2, characterized in that The performing knowledge distillation on the student language model according to the second text label and the second prediction result includes: Calculating the JS divergence of the teacher language model and the student language model according to the second text label and the second prediction result; Calculate the earth moving distance of the feature transfer of each layer in the teacher language model to each layer in the student language model; Calculating the loss between the teacher language model and the student language model according to the JS divergence and the earth mover distance; The student language model is updated according to the loss between the teacher language model and the student language model.

4. The method according to claim 1, characterized in that: The step of inputting the second task label into the teacher language model to generate a plurality of second predicted texts corresponding to the new task includes: The second task label is input into the teacher language model, and the plurality of second predicted texts are generated according to the number and a preset ratio of the second training texts in the second training text set.

5. A method for processing NLP tasks, characterized in that: include: Get the text for NLP tasks; The text is input into the student language model as described in any one of claims 1-4 to generate a prediction result of the NLP task.

6. The method according to claim 5, characterized in that The text includes question prompt information and questions of the NLP task; the inputting the text into the student language model according to any one of claims 1 to 4 to generate a prediction result of the NLP task includes: The question prompt information and the question are input into the student language model to generate an answer corresponding to the question.

7. A language model training device, characterized in that: include: An acquisition module is used to acquire a training sample set; the training sample set includes a first task label of a new task, multiple first training texts of the new task and a first text label of each of the first training texts, and a second task label of each of at least one old task, wherein the old task is an NLP task for which a language model has been trained, and the new task is an NLP task for which the language model has not been trained, the first task label is used to indicate the task type to which the new task belongs, the second task label is used to indicate the task type to which the old task belongs, the task types of the new task and the old task are different, and the first text label is used to indicate the true value corresponding to the first training text; A copy module, used for copying the language model to obtain a teacher language model, and using the language model as a student language model; A generating module, used for inputting the second task label into the teacher language model, generating a plurality of second training texts corresponding to the old task and a second text label for each of the second training texts, wherein the second text label is used for indicating a true value corresponding to the second training text; The generating module is further used to input the first task label, the second task label, the first training text and the second training text into the student language model, and generate a first predicted text corresponding to the first task label, a first predicted result corresponding to the first training text, a second predicted text corresponding to the second task label and a second predicted result corresponding to the second training text; A training module, used for training the student language model according to the first training text and the first predicted text, the second training text and the second predicted text, the first prediction result and the first text label, the second prediction result and the second text label; The first training text and the first text label constitute a question-answer format text, and the question-answer format text includes question prompt information and questions generated according to the question prompt information.

8. A NLP task processing device, characterized in that: include: Acquisition module, used to obtain text for NLP tasks; A generation module is used to input the text into the student language model as described in any one of claims 1-4 to generate a prediction result of the NLP task.

Images