A Chinese word sense disambiguation method based on prompt learning

Abstract

The present invention relates to a Chinese word sense disambiguation method based on prompt learning, and belongs to the field of computer technology. Most methods based on BERT adopt the form of using BERT to accommodate downstream tasks. First, the pre-trained model is applied to the downstream task, and then the performance is improved by fine-tuning the pre-trained language model. However, as the volume of pre-trained language models continues to increase, the hardware requirements and actual costs of fine-tuning them are also becoming more difficult. The Chinese BERT is based on characters. Most BERT-based word sense disambiguation models can only add MASK tags in units of characters in pre-trained MLM model tasks. The word is the average of the characters, and the expression of the word is not accurate enough. The present invention mainly solves the problems of low disambiguation efficiency in Chinese word sense disambiguation models, the need for a large amount of data annotation during supervised classification, and the under-utilization of BERT's own capabilities.

Description

Technical Field

[0001] The invention belongs to the field of computer technology and relates to a Chinese word sense disambiguation method based on prompt learning. Background Art

[0002] Word sense disambiguation refers to the process of determining the semantics of an ambiguous word based on the context when it appears in a specific context. As a basic task in the field of natural language processing, it is widely used in other downstream tasks, such as machine translation, information extraction and information retrieval. Therefore, solving language ambiguity is one of the important research tasks.

[0003] There are two main types of word sense disambiguation, namely supervised word sense disambiguation and knowledge-based word sense disambiguation. Supervised word sense disambiguation requires a large amount of training corpus with word sense annotations. In supervised word sense disambiguation methods, relevant annotation data is mainly used to train neural networks to eliminate ambiguity, by extracting some artificially designed features, and then training classifiers for each ambiguous word. Therefore, the effect of supervised is better than that based on knowledge base. The word sense disambiguation method based on knowledge base mainly extracts the contextual vocabulary of the context where the ambiguous word is located as the disambiguation feature, and constructs a large-scale knowledge base to obtain the relationship between the ambiguous word and the disambiguation feature, so as to determine the true meaning of the ambiguous word. The more commonly used knowledge bases include WordNet, Wikipedia, HowNet, etc. The word sense disambiguation method based on knowledge base can not only save costs and improve efficiency, but also avoid the problem of low disambiguation efficiency caused by the small size of the training corpus.

[0004] Insufficient existing technology:

[0005] (1) Most BERT-based methods use BERT to accommodate downstream tasks. First, the pre-trained model is applied to the downstream task, and then the performance is improved by fine-tuning the pre-trained language model. However, as the size of pre-trained language models continues to increase, the hardware requirements and actual cost of fine-tuning them are also becoming more difficult.

[0006] (2) BERT for Chinese is based on characters. Most BERT-based word sense disambiguation models can only add MASK tags on a character basis in the pre-trained MLM model task. The word is the average of the characters, which is not accurate enough in expressing the word.

[0007] The present invention mainly solves the problems of low disambiguation efficiency of the Chinese word sense disambiguation model, the need for large amounts of data annotation during supervised classification, and underutilization of BERT's own capabilities. Summary of the invention

[0008] In view of this, an object of the present invention is to provide a Chinese word sense disambiguation method based on prompt learning.

[0009] To address the problem in question (1) that supervised methods and knowledge-based methods each have their own advantages and disadvantages, the present invention uses HowNet as the knowledge base and BERT for supervised classification.

[0010] In response to question (2), the present invention uses a prompt learning method and utilizes the Mask mechanism of BERT's pre-training task MLM to better combine the calculation of synonym similarity with the context and obtain richer semantic information.

[0011] In order to achieve the above object, the present invention provides the following technical solutions:

[0012] A Chinese word sense disambiguation method based on prompt learning, the method comprising the following steps:

[0013] Step 1: Based on the dataset, for each ambiguous word x i , this word has N meanings in the knowledge base HowNet; for each of its meanings Find a synonym set for a group Then replace the ambiguous words with their corresponding synonyms to generate a new replacement sentence B;

[0014] Step 2: Replace the original ambiguous word with all synonyms corresponding to each meaning of the ambiguous word, obtain a new replacement sentence through data preprocessing, set labels, train the model, and complete supervised binary classification; determine whether the replaced sentence is consistent with the original sentence, and calculate the error with the given label;

[0015] For the input text x, there is a function f prompt (x), convert x into the form x′ of Prompt, as shown in formula (1);

[0016] x′=f prompt (x) (1)

[0017] Among them, combined with the pre-training task MLM of the pre-trained language model BERT, the following prompt is set: "The two sentences have the same meaning [MASK]: ContextA, ContextB;". ContextA is the original sentence, ContextB is the sentence after the synonyms are replaced, and [MASK] is the reserved answer blank;

[0018] By predicting the value of [MASK], we determine whether the meaning of the replacement sentence is similar to that of the original sentence, thereby obtaining the similarity label between the replacement sentence and the original sentence in each meaning of the ambiguous word;

[0019] Step 3: Through prompt learning, the correct and incorrect labels of the replacement sentences have been obtained; then the correct meaning of the ambiguous words in each sentence needs to be found; through the predicted label b of the replacement sentence i , the score of each meaning can be obtained through formula (2) The meaning with the highest score at the end is the final meaning;

[0020]

[0021] After calculating the similarity score of each word meaning, the similarity scores of the N word meanings in the ambiguous word are obtained, and the word meaning with the highest similarity score is taken as the final word meaning of the ambiguous word, as shown in formula (3):

[0022] y=MAX(G(S i |x)) (3).

[0023] Optionally, the data set is a Chinese word sense disambiguation data set OpenHowNet WSD based on HowNet constructed using the Chinese word sense annotation corpus in SemEval-2007#task5; the data set consists of 2967 word segmentation and part-of-speech tagging instance sentences, including 36 polysemous words, and the 36 polysemous words include 16 nouns and 20 verbs;

[0024] 2201 sentences were selected for training and 766 sentences were selected for testing;

[0025] Micro-F1 and Macro-F1 are used as evaluation indicators. The experimental environment is based on keras, and the Chinese Bert-Base-Chinese model is used as the encoder, with a total of 12 layers, 768 hidden units and 12 attention heads. The model sets the learning rate to 0.0006, the batch_size to 16, and Adam to optimize the model. The training model uses a TESLAV100-SXM2 graphics card with a memory of 32G.

[0026] A computer system comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the method according to any one of claims 1 to 2 when executing the computer program.

[0027] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 2.

[0028] The beneficial effects of the present invention are:

[0029] (1) In the similarity prediction module, a synonym set of word meaning is obtained based on HowNet, and a new replacement sentence is generated by replacing ambiguous words with synonyms. Then the similarity between the replacement sentence and the original sentence is calculated.

[0030] (2) When training the model for supervised classification, the pre-trained language model BERT is used not only to integrate the semantic information of the context, but also to improve the ability to predict the binary classification of similarity labels.

[0031] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0033] Figure 1 This is the structural diagram of the Chinese word sense disambiguation model based on prompt learning. DETAILED DESCRIPTION

[0034] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0035] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0036] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0037] See also Figure 1 The specific implementation steps of the present invention are as follows:

[0038] Step 1: For each ambiguous word x i , this word has N meanings in the knowledge base HowNet. For each of its meanings You can find a set of synonyms for a group Then replace the ambiguous words with their corresponding synonyms to generate a new replacement sentence B.

[0039] Step 2: Replace the original ambiguous word with all synonyms corresponding to each meaning of the ambiguous word, obtain a new replacement sentence through data preprocessing, set labels, train the model, and complete supervised binary classification. Determine whether the replaced sentence is consistent with the original sentence and calculate the error with the given label.

[0040] For the input text x, there is a function f prompt (x), convert x into the form x′ of Prompt, as shown in formula (1).

[0041] x′=f prompt (x) (1)

[0042] The Prompt set in the present invention is "Two sentences have the same meaning [MASK]: ContextA, ContextB." ContextA is the original sentence, and ContextB is the sentence after the synonyms are replaced.

[0043] The output of the designed Prompt is obtained using BERT's MLM task, and the output is "very" and "no". By predicting the value of [MASK], it is determined whether the meaning of the replacement sentence is similar to that of the original sentence, thereby obtaining the similarity label between the replacement sentence and the original sentence in each meaning of the ambiguous word.

[0044] Step 3: Through prompt learning, the correct and incorrect labels of the replacement sentences have been obtained. Then we need to find the correct meaning of the ambiguous words in each sentence. i , the score of each meaning can be obtained through formula (2) The meaning with the highest score at the end is the final meaning.

[0045]

[0046] After calculating the similarity score of each word meaning, the similarity scores of the N word meanings in the ambiguous word are obtained, and the word meaning with the highest similarity score is taken as the final word meaning of the ambiguous word, as shown in formula (3).

[0047] y=MAX(G(S i |x)) (3)

[0048] The dataset of the present invention is a Chinese word sense disambiguation dataset based on HowNet, OpenHowNet WSD, which is constructed based on the Chinese word sense annotation corpus in SemEval-2007#task5. The dataset consists of 2967 word segmentation and part-of-speech tagging example sentences, including 36 polysemous words (16 nouns and 20 verbs).

[0049] The present invention selects 2201 sentences for training and 766 sentences for testing. The division of the data set and the number of senses of ambiguous words are shown in Table 1.

[0050] Table 1 Composition of the dataset

[0051]

[0052] The present invention uses Micro-F1 and Macro-F1 as evaluation indicators. The experimental environment is based on keras, using the Chinese Bert-Base-Chinese model as the encoder, with a total of 12 layers, 768 hidden units and 12 attention heads. The model sets the learning rate to 0.0006, batch_size to 16, and uses Adam to optimize the model. The training model uses a 32G TESLA V100-SXM2 graphics card.

[0053] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0054] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0055] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the Chinese word sense disambiguation method and technique based on prompt learning described in the present invention, the present invention also includes the computer itself.

[0056] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A Chinese word sense disambiguation method based on prompt learning, characterized by: The method comprises the following steps: Step 1: Based on the dataset, for each ambiguous word x i , this word has N meanings in the knowledge base HowNet; for each of its meanings Find a synonym set for a group Then replace the ambiguous words with their corresponding synonyms to generate a new replacement sentence B; Step 2: Replace the original ambiguous word with all synonyms corresponding to each meaning of the ambiguous word, obtain a new replacement sentence through data preprocessing, set labels, train the model, and complete supervised binary classification; determine whether the replaced sentence is consistent with the original sentence, and calculate the error with the given label; For the input text x, there is a function f prompt (x), convert x into the form x′ of Prompt, as shown in formula (1); x′=f prompt (x) (1) Among them, combined with the pre-training task MLM of the pre-trained language model BERT, the following prompt is set: "The two sentences have the same meaning [MASK]: ContextA, ContextB;"; ContextA is the original sentence, ContextB is the sentence after the synonyms are replaced, and [MASK] is the reserved answer blank; The pre-training task MLM of the pre-trained language model BERT is used to obtain the output of the designed Prompt, which is "very" and "not". By predicting the value of [MASK], it is determined whether the meaning of the replacement sentence is similar to that of the original sentence, thereby obtaining the similarity label between the replacement sentence and the original sentence in each meaning of the ambiguous word. Step 3: Through prompt learning, the correct and incorrect labels of the replacement sentences have been obtained; then the correct meaning of the ambiguous words in each sentence needs to be found; through the predicted label b of the replacement sentence i , the score of each meaning can be obtained through formula (2) The meaning with the highest score at the end is the final meaning; After calculating the similarity score of each word meaning, the similarity scores of the N word meanings in the ambiguous word are obtained, and the word meaning with the highest similarity score is taken as the final word meaning of the ambiguous word, as shown in formula (3): y=MAX(G(S i |x)) (3)。 2. The Chinese word sense disambiguation method based on prompt learning according to claim 1, characterized in that: The dataset is a Chinese word sense disambiguation dataset OpenHowNet WSD based on HowNet, which is constructed based on the Chinese word sense annotation corpus in SemEval-2007#task5. The dataset consists of 2967 word segmentation and part-of-speech tagging instance sentences, including 36 polysemous words, including 16 nouns and 20 verbs. 2201 sentences were selected for training and 766 sentences were selected for testing; Micro-F1 and Macro-F1 are used as evaluation indicators. The experimental environment is based on keras, and the Chinese Bert-Base-Chinese model is used as the encoder, with a total of 12 layers, 768 hidden units and 12 attention heads. The model sets the learning rate to 0.0006, the batch_size to 16, and Adam to optimize the model. The training model uses a 32G TESLA V100-SXM2 graphics card.

3. A computer system comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 2 is implemented.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images