A processing method, system, electronic device and storage medium for NLU tasks

Abstract

The present application discloses a method for processing NLU tasks. The method includes: obtaining a pre-trained language model, where the pre-trained language model is a generative language model; receiving an NLU task and performing format conversion on the data content corresponding to the NLU task to obtain a target text, where the target text includes a text content sentence and a label sentence, the label sentence is the last sentence of the target text, and the label sentence is the sentence for filling in the label words of the text content sentence; inputting the target text into the pre-trained language model to obtain the label words of the text content sentence, and generating a processing result of the NLU task according to the label words of the text content sentence. The present application can improve the processing accuracy of the generative language model for NLU tasks. The present application also discloses an NLU task processing system, an electronic device, and a storage medium, which have the above beneficial effects.

Description

Technical Field

[0001] This application relates to the technical field of natural language processing, and particularly relates to a method, a system, an electronic device, and a storage medium for processing NLU tasks. Background Art

[0002] The rapid development of natural language processing technology has given birth to the pre-trained language model (PLM, Pre-trained Language Model). Among them, the relatively representative model structures are GPT and Bert. The emergence of pre-trained language models has, to some extent, reduced the usage threshold of natural language processing models because people only need to fine-tune the existing pre-trained models for their own downstream tasks they need, rather than training models from scratch, thereby greatly reducing the usage cost of natural language models. Since the developers of pre-trained language models do not know what downstream tasks the models will be used for, they often use a large amount of corpus to train a relatively large model to ensure that the models obtain the ability to perform different tasks.

[0003] Generative language models (such as GPT models) are pre-trained language models used to perform natural language generation tasks. Generative language models can perform zero-shot learning and few-shot learning without changing parameters, but the accuracy of using generative language models to process NLU tasks is relatively poor.

[0004] Therefore, how to improve the processing accuracy of generative language models for NLU tasks is a technical problem that those skilled in the art need to solve currently. Summary of the Invention

[0005] The purpose of this application is to provide a method, a system, an electronic device, and a storage medium for processing NLU tasks, which can improve the processing accuracy of generative language models for NLU tasks.

[0006] To solve the above technical problem, this application provides a method for processing NLU tasks, and the method for processing NLU tasks includes:

[0007] Obtain a pre-trained language model; wherein, the pre-trained language model is a generative language model;

[0008] Receive an NLU task, and perform format conversion on the data content corresponding to the NLU task to obtain a target text; wherein, the target text includes a text content sentence and a label sentence, the label sentence is the last sentence of the target text, and the label sentence is the sentence for filling in the label words of the text content sentence;

[0009] Input the target text into the pre-trained language model to obtain the labeled words of the text content statement, and generate the processing result of the NLU task according to the labeled words of the text content statement.

[0010] Optionally, inputting the target text into the pre-trained language model to obtain the labeled words of the text content statement includes:

[0011] Input the target text into the pre-trained language model to obtain the predicted labeled word token filled by the pre-trained language model in the labeled statement;

[0012] Determine whether the predicted labeled word token is the standard labeled word of the text content statement;

[0013] If so, set the predicted labeled word token as the labeled word of the text content statement.

[0014] Optionally, after determining whether the predicted labeled word token is the standard labeled word of the text content statement, it further includes:

[0015] If the predicted labeled word token is not the standard labeled word of the text content statement, use the pre-trained language model to calculate the probability of each alternative labeled word in the labeled word set appearing in the labeled statement;

[0016] Set the alternative labeled word with the highest probability as the labeled word of the text content statement.

[0017] Optionally, the format conversion of the data content corresponding to the NLU task to obtain the target text includes:

[0018] Determine the text content statement and the standard labeled word in the data content;

[0019] Construct a target statement including the standard labeled word, and delete the standard labeled word in the target statement to obtain the labeled statement;

[0020] Construct the target text including the text content statement and the labeled statement.

[0021] Optionally, the construction of the target statement including the standard labeled word includes:

[0022] Construct the target statement including the standard labeled word and the standard labeled word is at the end of the sentence.

[0023] Optionally, the pre-trained language model is specifically a pre-trained language model with a GPT structure.

[0024] Optionally, after generating the processing result of the NLU task according to the label words of the text content sentence, the following steps are further included:

[0025] Transmit the processing result of the NLU task to the downstream task.

[0026] This application also provides a processing system for NLU tasks, which includes:

[0027] A model acquisition module, configured to acquire a pre-trained language model; wherein, the pre-trained language model is a generative language model;

[0028] A format conversion module, configured to receive an NLU task and perform format conversion on the data content corresponding to the NLU task to obtain a target text; wherein, the target text includes a text content sentence and a label sentence, the label sentence is the last sentence of the target text, and the label sentence is the sentence for filling in the label words of the text content sentence;

[0029] A processing module, configured to input the target text into the pre-trained language model to obtain the label words of the text content sentence, and generate a processing result of the NLU task according to the label words of the text content sentence.

[0030] This application also provides a storage medium, on which a computer program is stored, and when the computer program is executed, the steps executed by the above-mentioned NLU task processing method are implemented.

[0031] This application also provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps executed by the above-mentioned NLU task processing method are implemented.

[0032] This application provides a method for processing NLU tasks, including: acquiring a pre-trained language model; wherein, the pre-trained language model is a generative language model; receiving an NLU task and performing format conversion on the data content corresponding to the NLU task to obtain a target text; wherein, the target text includes a text content sentence and a label sentence, the label sentence is the last sentence of the target text, and the label sentence is the sentence for filling in the label words of the text content sentence; inputting the target text into the pre-trained language model to obtain the label words of the text content sentence, and generating a processing result of the NLU task according to the label words of the text content sentence.

[0033] After receiving the NLU task, this application performs format conversion on the data content corresponding to the NLU task to obtain the target text. The target text includes a text content statement and a tag statement. The tag statement is the last sentence of the target text, and the tag statement is the sentence for filling in the tag words of the text content statement. After inputting the target text into the pre-trained language model, it is equivalent to converting the NLU task into a natural language generation task that the generative language model is good at processing. Therefore, this application can improve the processing accuracy of the generative language model for the NLU task. This application also provides a processing system for the NLU task, an electronic device, and a storage medium, which have the above beneficial effects and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] To more clearly illustrate the embodiments of this application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0035] Figure 1 It is a flowchart of a method for processing the NLU task provided by an embodiment of this application;

[0036] Figure 2 It is a schematic structural diagram of a processing system for the NLU task provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the drawings in the embodiments of this application. Obviously, the described embodiments are some, but not all, of the embodiments of this application. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

[0038] Please refer to the following Figure 1 , Figure 1 It is a flowchart of a method for processing the NLU task provided by an embodiment of this application.

[0039] The specific steps may include:

[0040] S101: Obtain a pre-trained language model;

[0041] Among them, this embodiment can be applied to an intelligent dialogue system. The above pre-trained language model is a generative language model, and a generative language model is a model capable of performing natural language generation tasks (Natural Language Generating). Specifically, the above generative language model is generally a pre-trained language model with a GPT (Generative Pre-training Transformer) structure.

[0042] S102: Receive an NLU task, and perform format conversion on the data content corresponding to the NLU task to obtain a target text.

[0043] Among them, NLU (Natural Language Understanding) is a natural language understanding task. After receiving the natural language understanding task, this embodiment performs format conversion on the data content to obtain a target text, so as to convert the natural language understanding task into a natural language generation task. The target text in this embodiment includes a text content sentence and a label sentence. The label sentence is the last sentence of the target text, and the label sentence is a sentence for filling in the label word of the text content sentence. Specifically, this embodiment can determine the label sentence according to the semantics of the target text, and this process can be implemented by a staff member.

[0044] Specifically, this embodiment can obtain the target text in the following way: determine the text content sentence and the standard label word in the data content; construct a target sentence including the standard label word; construct the target text including the text content sentence and the label sentence. Specifically, this embodiment can construct the target sentence including the standard label word and the standard label word is at the end of the sentence.

[0045] For example, the data content corresponding to the NLU task is: "The weather tomorrow is sunny. It is recommended to open the window for ventilation. Please pay attention to sun protection.", the text content sentence is "Tomorrow is sunny. It is recommended to open the window for ventilation. Please pay attention to sun protection.", the standard label word is "weather", and the target sentence constructed including the standard label word and the standard label word is at the end of the sentence is: "The theme of this message is weather", delete the standard label word in the target sentence to obtain the label sentence "The theme of this message is __", and the finally constructed target text is "Tomorrow is sunny. It is recommended to open the window for ventilation. Please pay attention to sun protection. The theme of this message is _".

[0046] S103: Input the target text into the pre-trained language model to obtain the label word of the text content sentence, and generate the processing result of the NLU task according to the label word of the text content sentence.

[0047] Among them, in this embodiment, the label words of the text content sentence can be obtained in the following manner: input the target text into the pre-trained language model to obtain the predicted label word token filled in by the pre-trained language model in the label sentence; determine whether the predicted label word token is the standard label word of the text content sentence; if so, set the predicted label word token as the label word of the text content sentence; if not, calculate the probability of each alternative label word in the label word set appearing in the label sentence by using the pre-trained language model, and then set the alternative label word with the highest probability as the label word of the text content sentence.

[0048] After receiving the NLU task in this embodiment, the data content corresponding to the NLU task is format-converted to obtain the target text. The target text includes a text content sentence and a label sentence. The label sentence is the last sentence of the target text, and the label sentence is the sentence for filling in the label word of the text content sentence. After inputting the target text into the pre-trained language model, it is equivalent to converting the NLU task into a natural language generation task that the generation-type language model is good at processing. Therefore, this embodiment can improve the processing accuracy of the generation-type language model for the NLU task.

[0049] As a feasible implementation manner, after generating the processing result of the NLU task according to the label word of the text content sentence, the processing result of the NLU task can also be passed to the downstream task. The downstream task can be a task of returning the response content corresponding to the processing result.

[0050] The following uses an embodiment in actual application to illustrate the process described in the above embodiment.

[0051] When pre-trained language models are applied to downstream tasks, different types of pre-trained language models have different characteristics. Taking the Bert and GPT architectures as examples: When using Bert, it is usually fine-tuned for a specific downstream task to adjust the relevant parameters in the model, so as to achieve excellent performance on the downstream task. The feature extraction structure of Bert's bidirectional transformer helps to effectively extract context information, and the pre-training method of Masked Language Model makes it easier to achieve good results in NLU tasks. However, when encoding sequences with the help of context, it is also easy for the data in the pre-training process to be mismatched with the data in fine-tuning, and generally it is difficult to adapt to generative tasks. In practical applications, Bert can be used for sentiment analysis of customer reviews and assist search based on existing information. GPT, on the other hand, is a unidirectional generative pre-trained model that uses multiple layers of transformer decoders for feature extraction. As an autoregressive language model, the model structure of GPT is naturally suitable for processing generative language tasks (NLG) tasks. However, because it cannot consider the context information comprehensively, its performance in NLU tasks will be relatively weak, especially in tasks such as judging the relationship between two sentences and summarizing text content. The model with GPT architecture can also be fine-tuned for downstream tasks. However, compared with models of other architectures, the advantage of GPT is that it can perform zero-shot learning and few-shot learning without changing parameters. Since no fine-tuning is required, the usage cost of the model can be further reduced. In practical applications, GPT is usually used for text generation-related work, such as machine translation, document generation, dialogue generation, etc.

[0052] Due to the advantage of zero-shot learning of the model with GPT architecture, the model provided in this embodiment can not only be used for generative tasks, but also has a method to overcome the weaknesses of its structure and apply the idea of NLG tasks to the solution of NLU tasks. NLU tasks are the general term for all method tasks that support machines to understand text content. Generally speaking, they include tasks such as word segmentation, syntactic analysis, text classification, information retrieval, information extraction, text proofreading, text clustering, etc. These tasks usually focus on the understanding of text semantics and basically do not require additional text generation. They can also be regarded as the basis of text generation tasks.

[0053] In the existing solutions, when the GPT model processes NLU tasks, it adopts the method of pre-training + fine-tuning: in the first stage, a large amount of unlabeled corpus is used for unsupervised pre-training, and the loss of the language model is used; in the second stage, for different tasks, a linear transformation layer is added after the pre-trained model, and fine-tuning is performed by combining the losses of different tasks. This solution was the solution for NLU tasks in the initial stage when the GPT model was born. This solution also followed the processing method of pre-trained language models before, and it was necessary to adjust the model parameters in both steps of training. However, the disadvantage of the above solution is that it is necessary to adjust the model parameters in both stages of training. That is to say, after the user obtains the pre-trained model, when performing downstream tasks, it is still necessary to consume computing resources to perform the second-stage training of the model, and this consumption of computing resources cannot be ignored.

[0054] The GPT model can perform downstream tasks through zero-shot learning, that is, without updating the network parameters, and obtain the desired result by only describing the task. After the user obtains the pre-trained model, they can no longer perform the next step of training, which can save the computing resources of this step. At present, there are many works trying to use GPT zero-shot learning to solve NLU tasks, but most of them do not introduce the specific solution of zero-shot learning in detail, and GPT is essentially still a generative model. Most of the current works on zero-shot learning in downstream tasks do not utilize the generative characteristics of the GPT model, which is a waste of the text generation ability of this model.

[0055] In order to overcome the defects existing in the above-mentioned prior art, the present application provides a solution for converting natural language understanding tasks into natural language generation tasks. This solution can apply the text generation ability of the GPT model to NLU downstream tasks, thereby improving the accuracy in NLU tasks without adding additional model parameters. This solution can also not fine-tune the model parameters when using GPT for downstream tasks, reducing the consumption of computing resources.

[0056] The basic idea of this embodiment is to use the text generation ability of GPT to solve NLU problems. During the process of solving the problem, the pre-trained model with the GPT structure is directly used, and the model structure or parameters are not changed for downstream tasks; the text is rewritten according to the task characteristics of NLU, and the position of the label is left at the end of the sentence; the rewritten text is input into the model, and the model is made to generate a token; finally, the generated token is compared with the label. This embodiment may include the following steps:

[0057] Step 1: Obtain the pre-trained model with the GPT structure and perform zero-shot learning

[0058] Step 2: Data processing.

[0059] This step can analyze the text content and label characteristics of the NLU task. When processing the text format, leave a space for the label at the end of the last token. If the length of the label is greater than 1 token, convert the label to 1 token. This step rewrites the data of the NLU task after adding prompt words, places the statement in the front, writes the last sentence in the form of a half sentence, and finally leaves a space for 1 token for the label.

[0060] Taking the sentiment classification task as an example: The dataset includes sentence S1, and labels 0 (negative) and 1 (positive). When processing the data, we will add some prompt words and rewrite the statement and label into the form of "S1, this restaurant is very _____", leaving the last position empty.

[0061] Taking the task of judging whether the semantics between two sentences are consistent as an example: The dataset includes sentences S1 / S2, and labels 0 (different) and 1 (same). When processing the data, rewrite the statement and label into the form of "S1. Question: S2. The meaning of the two sentences is _____", leaving the last position empty. The same applies to other NLU tasks. The information of the statement is placed in the front, the last sentence is set in the form of a sentence to be completed, and the empty part that needs to be completed is the position of the label (i.e., the answer).

[0062] Step 3: Input the data processed in the previous step into the pre-trained model, and let the model generate a token to complete the last sentence. The algorithm used to generate this token remains unchanged. At the same time, calculate the probability that the generated token and all label words will appear in this position.

[0063] Step 4: If the token generated by the model is one of the labels, the answer is that token; if not, select the one with the highest probability of appearing here among all label words as the answer.

[0064] The above process makes the model generate a token. If one of the labels is directly generated, that token is the answer predicted by the model; if not, compare the probabilities that all label words may appear here, and take the label word with the largest probability value as the answer predicted by the model.

[0065] The above embodiments can reasonably convert the dataset of the NLU task into a form that is convenient for processing by the GPT architecture model, thereby converting the NLU task that the GPT architecture model is not good at into an NLG task, effectively improving the accuracy of the NLU task. When the model cannot generate the label word we want, the task can also be converted into a comparison of the occurrence probabilities of the label words, ensuring that the zero-shot learning ability of the GPT model can be utilized to the maximum extent, and avoiding changing the model structure or fine-tuning when processing the NLU task, thereby avoiding additional consumption of computing resources. The above embodiments can also be applied to other autoregressive language models similar to the GPT architecture, and have the same technical effects.

[0066] Please refer to Figure 2 , Figure 2 which is a schematic structural diagram of a processing system for NLU tasks provided by an embodiment of the present application. The system may include:

[0067] A model acquisition module 201, configured to acquire a pre-trained language model; wherein, the pre-trained language model is a generative language model;

[0068] A format conversion module 202, configured to receive an NLU task and perform format conversion on the data content corresponding to the NLU task to obtain a target text; wherein, the target text includes a text content statement and a label statement, the label statement is the last sentence of the target text, and the label statement is a sentence for filling in the label word of the text content statement;

[0069] A processing module 203, configured to input the target text into the pre-trained language model to obtain the label word of the text content statement, and generate a processing result of the NLU task according to the label word of the text content statement.

[0070] In this embodiment, after receiving the NLU task, the data content corresponding to the NLU task is subjected to format conversion to obtain a target text. The target text includes a text content statement and a label statement. The label statement is the last sentence of the target text, and the label statement is a sentence for filling in the label word of the text content statement. After inputting the target text into the pre-trained language model, it is equivalent to converting the NLU task into a natural language generation task that the generative language model is good at processing. Therefore, this embodiment can improve the processing accuracy of the generative language model for the NLU task.

[0071] Further, the processing module 203 is configured to input the target text into the pre-trained language model to obtain the predicted label word token filled by the pre-trained language model in the label statement; and is further configured to determine whether the predicted label word token is the standard label word of the text content statement; if so, set the predicted label word token as the label word of the text content statement; if not, calculate the probability of each alternative label word in the label word set appearing in the label statement by using the pre-trained language model, and set the alternative label word with the highest probability as the label word of the text content statement.

[0072] Further, the format conversion module 202 is configured to determine the text content statement and the standard label word in the data content; is further configured to construct a target statement including the standard label word, and delete the standard label word in the target statement to obtain the label statement; and is further configured to construct the target text including the text content statement and the label statement.

[0073] Further, the process of the format conversion module 202 constructing the target statement including the standard label word includes: constructing the target statement including the standard label word and the standard label word being at the end of the sentence.

[0074] Further, the pre-trained language model is specifically a pre-trained language model with a GPT structure.

[0075] Further, it further includes:

[0076] A result transmission module, configured to transmit the processing result of the NLU task to a downstream task after generating the processing result of the NLU task according to the label word of the text content statement.

[0077] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, and details are not described here for the moment.

[0078] This application also provides a storage medium, on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiments can be implemented. The storage medium may include: various media capable of storing program codes such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.

[0079] The present application further provides an electronic device, which may include a memory and a processor. A computer program is stored in the memory. When the processor calls the computer program in the memory, the steps provided in the above embodiments can be implemented. Of course, the electronic device may further include various network interfaces, power supplies and other components.

[0080] The various embodiments in the specification are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part. It should be noted that for those of ordinary skill in the art in the technical field of the present application, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

[0081] It should also be noted that in this specification, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or device including the said element.

Claims

1. A processing method for NLU tasks, characterized in that, Including: Obtain a pre-trained language model; wherein, the pre-trained language model is a generative language model; Receive an NLU task, and perform format conversion on the data content corresponding to the NLU task to obtain a target text; wherein, the target text includes a text content sentence and a label sentence, the label sentence is the last sentence of the target text, the label sentence is the sentence for filling in the label word of the text content sentence, and moreover, the label sentence is determined based on the semantics of the target text; Input the target text into the pre-trained language model to obtain the label word of the text content sentence, and generate a processing result of the NLU task according to the label word of the text content sentence; Wherein, the performing format conversion on the data content corresponding to the NLU task to obtain a target text includes: Determine the text content sentence and the standard label word in the data content; Construct a target sentence including the standard label word, and delete the standard label word in the target sentence to obtain the label sentence; Construct the target text including the text content sentence and the label sentence.

2. The processing method of the NLU task according to claim 1, wherein Inputting the target text into the pre-trained language model to obtain the label word of the text content sentence includes: Input the target text into the pre-trained language model to obtain the predicted label word token filled in by the pre-trained language model in the label sentence; Judge whether the predicted label word token is the standard label word of the text content sentence; If so, set the predicted label word token as the label word of the text content sentence.

3. The processing method of the NLU task according to claim 2, wherein After judging whether the predicted label word token is the standard label word of the text content sentence, it further includes: If the predicted label word token is not the standard label word of the text content sentence, calculate the probability of each alternative label word in the label word set appearing in the label sentence by using the pre-trained language model; Set the alternative label word with the highest probability as the label word of the text content sentence.

4. The processing method of the NLU task according to claim 1, wherein The constructing a target sentence including the standard label word includes: Construct the target sentence including the standard label word and the standard label word is at the end of the sentence.

5. The processing method of the NLU task according to claim 1, characterized in that The pre-trained language model is specifically a pre-trained language model with a GPT structure.

6. The processing method of the NLU task according to any one of claims 1 to 5, characterized in that, After generating the processing result of the NLU task according to the label word of the text content sentence, it further includes: Transmit the processing result of the NLU task to the downstream task.

7. A processing system for NLU tasks, characterized in that, Including: A model acquisition module, configured to obtain a pre-trained language model; wherein, the pre-trained language model is a generative language model; A format conversion module, configured to receive an NLU task, and perform format conversion on the data content corresponding to the NLU task to obtain a target text; wherein, the target text includes a text content sentence and a label sentence, the label sentence is the last sentence of the target text, the label sentence is the sentence for filling in the label word of the text content sentence, and moreover, the label sentence is determined based on the semantics of the target text; A processing module, configured to input the target text into the pre-trained language model to obtain the labeled words of the text content statement, and generate a processing result of the NLU task according to the labeled words of the text content statement; Wherein, the format conversion module is specifically configured to: Determine the text content statement and the standard labeled words in the data content; Construct a target statement including the standard labeled words, and delete the standard labeled words in the target statement to obtain the labeled statement; Construct the target text including the text content statement and the labeled statement.

8. An electronic device, characterized in that, Comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the processing method of the NLU task according to any one of claims 1 to 6 are implemented.

9. A storage medium, characterized in that, Computer-executable instructions are stored in the storage medium, and when the computer-executable instructions are loaded and executed by a processor, the steps of the processing method of the NLU task according to any one of claims 1 to 6 are implemented.

Images