A Long-Tail Word Sense Disambiguation Method Integrating Definition and Scenario Matching Mechanisms

Abstract

The present invention discloses a long-tail word sense disambiguation method integrating a definition and a scenario matching mechanism, including: learning a text embedding of a word sense definition (i.e., a definition embedding) from a word sense annotation text in a dictionary, wherein a text-to-vector mapping model is implemented by a definition encoder; learning a text embedding of a word sense scenario (i.e., a scenario embedding) from a word sense example sentence text in the dictionary, wherein a text-to-vector mapping model is implemented by a scenario encoder; in the process of obtaining the scenario embedding, hiding the target word contained in the input text to obtain the scenario information of the word sense; learning a word embedding of the target word (i.e., a target word embedding) from a text to be disambiguated containing the target word; the pre-trained language model for obtaining the target word embedding and the pre-trained language model for obtaining the scenario embedding share one; and superimposing and summing the probabilities of the calculation results under the definition matching mechanism and the scenario matching mechanism as the final output value to determine the final word sense of the target word.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and machine learning, and particularly to a long-tail word sense disambiguation method integrating a definition and a scenario matching mechanism. Background Art

[0002] With the continuous development of artificial intelligence technology, the information processing capabilities and levels of intelligent systems have made great progress, and their practicality and ease of use have also been recognized by end users. Among intelligent products based on human-computer dialogue technology, intelligent products such as Siri of Apple Inc., Now of Google Inc., Xiaoice of Microsoft Corporation, Xiaodu of Baidu Inc., and Xiaomi of Alibaba Group have been accepted by the majority of consumers; among intelligent products based on machine translation technology, intelligent products such as Google Translate of Google Inc., Bing Translator of Microsoft Corporation, TMT of Tencent Corporation, and Baidu Translate of Baidu Inc. have also been applied in various fields of our lives.

[0003] Natural Language Processing (NLP) is a discipline that takes natural language as the research object and uses machine learning technology to analyze, understand, and process natural language. It plays an important role in tasks that require natural language processing such as human-computer dialogue and machine translation. Word Sense Disambiguation (WSD) aims to determine the word sense to which a target word belongs based on the given context, and belongs to a fundamental research topic in the field of natural language processing. It has a crucial position in tasks that require natural language understanding such as human-computer dialogue, machine translation, and text generation. Relevant research has confirmed that having a clear word sense recognition of the basic components of text (i.e., vocabulary) is beneficial to the accurate processing of subsequent high-order tasks.

[0004] However, word sense disambiguation is a difficult task. It not only requires the context in which the target word is located to provide effective information for distinguishing word senses, but also requires each word sense of the target word to have a clear and distinguishable definition. That is to say, insufficient context information or vague word sense definitions will cause errors in word sense disambiguation. The root cause of vague word sense definitions is that humans do not always reach an agreement on all word senses, that is, the concept of "word sense" itself is ambiguous and controversial. For the problem of insufficient context information, an effective solution is to expand the scope of the context or make the most likely prediction based on a large amount of known data; but for the problem of vague word sense definitions, there is no widely recognized solution. It should be emphasized that: high-frequency word senses (or so-called head word senses) rarely exhibit the phenomenon of vague word sense definitions; low-frequency word senses (or so-called long-tail word senses) are the hardest hit areas of this phenomenon. This conclusion also conforms to common sense, because humans usually try their best to avoid language ambiguity, so they will correspondingly reduce the use of word senses with vague definitions. Therefore, long-tail word sense disambiguation is the main challenge faced by current word sense disambiguation.

[0005] Currently, the ideas of researchers can be summarized into methods of data augmentation, knowledge transfer, and imposing external constraints. The method of data augmentation attempts to expand the amount of text for word sense definitions to improve or enhance the accuracy of word sense representation. The most effective method among them is to integrate word sense definitions in different languages. The method of knowledge transfer attempts to fuse external knowledge to strengthen the accuracy of word sense representation. The most concerned method among them is to directly integrate the knowledge of the context where the word sense applies to improve the accuracy of word sense representation. It should be emphasized that since the external knowledge in this method directly comes from the context in the same semantic space of the target word, this method significantly improves the accuracy of word sense recognition. The method of imposing constraints assumes that the representation of word sense definitions is not a vector in a discrete space, but in a continuous space. Therefore, this method obtains or improves the word sense representation of vague word sense definitions through known or determined word sense representations. However, all of the above methods implicitly assume that all word senses can be clearly and distinguishablely defined. Linguists have confirmed that humans do not always reach a consensus on all word sense definitions. In other words, the word sense definitions obtained by the above methods are applicable in the corresponding scenarios, but it is difficult to effectively transfer to other scenarios. Summary of the Invention [[ID=|6]]

[0006] The object of the present invention is to overcome the deficiencies in the prior art and provide a long-tail word sense disambiguation method that integrates a definition and a scenario matching mechanism.

[0007] This method adopts a dual matching mechanism (i.e., definition matching and scenario matching) to handle the high-frequency word senses (or head word senses) with a huge sample size and the long-tail word senses (or tail word senses) with scarce samples in the word sense disambiguation task. Among them, the definition matching mechanism is a commonly used word sense recognition method in the word sense disambiguation task; while the scenario matching mechanism is a unique word sense recognition method proposed by the present invention for long-tail word senses; at the same time, the adoption of the dual recognition method is also unique to the present invention compared with traditional methods. This method makes up for the deficiency of the single definition recognition method commonly adopted in the implementation scheme of the word sense disambiguation system when dealing with long-tail word senses. In the implementation process of the word sense disambiguation system, the two matching mechanisms do not specifically distinguish between high-frequency word senses and long-tail word senses, but experimental analysis finds that the definition matching mechanism is good at recognizing high-frequency word senses, while the scenario matching mechanism is more suitable for long-tail word senses. The word sense disambiguation system constructed by this method is verified under the standard English full-word word sense disambiguation evaluation framework, and the experimental results show that this method obtains excellent performance under multiple evaluation sets.

[0008] The object of the present invention is achieved through the following technical solutions:

[0009] A long-tail word sense disambiguation method integrating definition and scenario matching mechanisms, comprising:

[0010] Learning the text embedding of the word sense definition from the word sense annotation text in the dictionary, that is, the definition embedding, where the text-to-vector mapping model uses the pre-trained language model BERT and is called the definition encoder Definition-BERT;

[0011] Learning the text embedding of the word sense scenario from the word sense example sentences text in the dictionary, that is, the scenario embedding, where the text-to-vector mapping model uses the pre-trained language model BERT and is called the scenario encoder Context-BERT; in the process of obtaining the scenario embedding, the target word (the word to be disambiguated) contained in the input text is hidden to obtain the scenario information of the word sense;

[0012] Learning the word embedding of the target word from the text to be disambiguated containing the target word, that is, the target word embedding, where the text-to-vector mapping model uses the pre-trained language model BERT; the pre-trained language model for obtaining the target word embedding and the pre-trained language model for obtaining the scenario embedding share a BERT model;

[0013] Based on the target word embedding and the definition embeddings of each word sense in the word sense list corresponding to the target word in the dictionary, calculating the similarity between the target word embedding and each definition embedding respectively to obtain the scores of each word sense in the word sense list under the definition matching mechanism;

[0014] Based on the target word embedding and the scene embedding of each word meaning in the word sense list corresponding to the target word, the similarity between the target word embedding and each scene embedding is calculated to obtain the score of each word meaning in the word sense list under the scene matching mechanism;

[0015] The probabilities of the scores calculated under the definition matching mechanism and the scene matching mechanism are superimposed and used as the final output value to determine the final meaning of the target word.

[0016] Furthermore, obtaining the definition embedding specifically includes:

[0017] (101) Encoding the word meaning definition text: adding a start tag [CLS] and a segmentation tag [SEP] at the beginning and end of the word meaning definition text respectively, and then using the mapping method BertTokenizer(·) of the BERT model to encode the words in the word meaning annotation text after adding the tags into corresponding numbers to obtain the encoded text of the word meaning annotation text;

[0018] (102) Vectorize the encoded text to obtain definition embeddings: The word embeddings of the corresponding numbers in the encoded text are obtained through the vectorization method Bert(·) of the BERT model, that is, the word embeddings of the corresponding words in the semantic annotation text.

[0019] Furthermore, it is characterized in that

[0020] For the word meaning annotation text W gloss ,

[0021] W gloss =[w1, w2, ..., w i， ...] (1)

[0022] where w i Refers to the words that make up the glossary text. The subscript gloss is intentionally omitted for clarity.

[0023] After adding the start and split markers:

[0024] W gloss =[[CLS],w i ,w2,...,w i ,...,[SEP]]. (2)

[0025] The mapping method BertTokenizer(·) of the BERT model is used to encode the words in the semantic annotation text into corresponding numbers to obtain the encoded text C of the semantic annotation text. gloss ,

[0026] C gloss =BertTokenizer(W gloss ). (3)

[0027] Obtain the encoded text C through the vectorization method Bert(·) of the BERT model gloss the word embeddings corresponding to each number in, that is

[0028]

[0029] wherein and respectively represent the word embeddings corresponding to the start and segmentation markers in the encoded text.

[0030] The corresponding word embeddings will be used as the definition embeddings of the sense annotation text, that is

[0031]

[0032] Furthermore, obtaining the scene embedding specifically includes:

[0033] (201) Encode the sense example sentence text: Add the start marker [CLS] and the segmentation marker [SEP] at the beginning and end of the sense example sentence text respectively, and the target word in the sense example sentence text is replaced with the mask marker [MASK]; then use the mapping method BertTokenizer(·) of the BERT model to encode the words in the sense example sentence text after adding the markers into corresponding numbers to obtain the encoded text of the sense example sentence text;

[0034] (202) Vectorize the encoded text to obtain the scene embedding: Obtain the word embeddings corresponding to the numbers in the encoded text through the vectorization method Bert(·) of the BERT model.

[0035] Furthermore, when the corresponding sense example sentence text in the dictionary does not exist, the disambiguation text to be processed is default selected as the example sentence for extracting scene information; when the number of corresponding sense example sentence texts in the dictionary is more than one, the example sentence with the largest number of words in the example sentence text is default selected as the example sentence text for extracting scene information; for the sense example sentence text W example ,

[0036] W example =[w1, w2,..., w target ,..., w i ,...](7)

[0037] where w i refers to the word constituting the sense example sentence text, and the subscript example is deliberately omitted for clarity of expression; w target represents the target word in the sense example sentence text, and it can appear at any position.

[0038] The form after adding the start and segmentation markers and replacing the mask marker is:

[0039] W example = [[CLS], w1, w2,..., [MASK],..., w i ,..., [SEP]]. (8)

[0040] Then, using the encoding method BertTokenizer(·) of the BERT model, the words in the sense example text are encoded into corresponding numbers, that is

[0041] C example = BertTokenizer(W example ). (9)

[0042] Based on the encoded text of the sense example text, the word embeddings corresponding to the numbers in the sense example text are obtained through the vectorization method Bert(·) of the BERT model:

[0043]

[0044] Among them and respectively represent the word embeddings corresponding to the start and end markers in the encoded text; v [MASK] represents the word embedding corresponding to the mask marker.

[0045] The corresponding word embeddings are used as the scene embeddings corresponding to the corresponding senses, that is

[0046]

[0047] Furthermore, obtaining the target word embeddings specifically includes:

[0048] (301) Encode the text to be disambiguated containing the target word: Add the start marker [CLS] and the segmentation marker [SEP] at the beginning and end of the text to be disambiguated containing the target word. For the text to be disambiguated W text ,

[0049] W text = [w1, w2,..., w i ,...]. (13)

[0050] where w i refers to the words that make up the text to be disambiguated, and the subscript text is deliberately omitted for clarity of expression; the target word w target belongs to W text , that is, w target ∈W text .

[0051] After adding the start and segmentation markers, it becomes:

[0052] W text = [[CLS], w1, w2,..., w i ,..., [SEP]]. (14)

[0053] Then, use the mapping method BertTokenizer(·) of the BERT model to encode the words in W text into corresponding numbers, that is

[0054] C text = BertTokenizer(W text ). (15)

[0055] (302) Vectorize the encoded text to obtain the target word embedding: Obtain the word embedding corresponding to the number in the encoded text through the vectorization method Bert(·) of the BERT model:

[0056]

[0057] Among them and respectively represent the word embeddings corresponding to the start and end markers in the encoded text; v target The corresponding word embedding is the word embedding of the target word.

[0058] Here, the word embedding v target corresponding to the target word is used as the final output target word embedding V target , that is

[0059] V target ≡ v target . (18)

[0060] Furthermore, define the probability superposition sum of the calculation results under the matching mechanism and the scenario matching mechanism as the final output value to determine the final meaning of the target word, specifically including:

[0061] (401) Calculate the similarity between the target word embedding and each definition embedding in the list of definitions corresponding to the target word:

[0062] Based on the obtained target word embedding V target and all the definition embeddings in the list of definitions corresponding to the target word where the index i refers to the i-th meaning in the list of definitions, and the score of the target word belonging to each meaning under the definition matching mechanism is

[0063]

[0064] where the operator ⊙ represents the vector dot product operation;

[0065] (402) Calculate the similarity between the target word embedding and each scenario embedding in the sense list corresponding to the target word:

[0066] Based on the target word embedding V target and the scenario embeddings of each sense in the sense list corresponding to the target word where the index i refers to the i-th sense in the sense list; under the scenario matching mechanism, the scores of the target word belonging to each sense are

[0067]

[0068] (403) Perform a probabilistic superposition summation on the scores of each sense obtained under the definition matching mechanism and the scenario matching, and use it as the final output value Score i , where the sense corresponding to the maximum output value is the most likely sense of the target word in the text to be disambiguated:

[0069]

[0070] where and α + β = 1; α, β are set as fixed values or variables learned based on other parameters.

[0071] In addition, during the model training process, the optimizer is the mainstream learning rate adaptive optimization algorithm Adam, and its loss function is

[0072]

[0073] where the index index represents the index of the sense in the sense list corresponding to the target word.

[0074] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the long-tail sense disambiguation method that fuses the definition and scenario matching mechanisms.

[0075] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the long-tail sense disambiguation method that fuses the definition and scenario matching mechanisms.

[0076] Compared with the prior art, the beneficial effects brought by the technical solution of the present invention are:

[0077] 1. Traditional definition matching mechanisms can effectively handle high-frequency word sense disambiguation, but are difficult to effectively handle long-tail word senses, which have serious phenomena of fixed collocations and specific applicable scenarios. The present invention utilizes the scenario information of long-tail word senses to enhance the word sense recognition ability, and proposes a scenario matching mechanism applicable to long-tail word sense recognition to address long-tail word sense disambiguation. This method takes into account the characteristics of long-tail word senses, namely, the collocation words and applicable scenarios are relatively fixed and it is difficult to form a unified and effective definition description. It uses the example sentences of word senses in the dictionary to learn the scenario embedding of the target word, and then realizes long-tail word sense disambiguation.

[0078] 2. During the implementation process, the definition matching and scenario matching methods are used in combination, and their coordination is achieved by using weight coefficients, ultimately realizing the effective recognition of all word senses including long-tail word senses. The word sense disambiguation method integrating the definition and scenario matching mechanisms makes up for the deficiencies of the word sense disambiguation system implemented by traditional single matching mechanisms in long-tail word sense disambiguation, and effectively improves the overall performance of the word sense disambiguation system.

[0079] 3. During the model training process, the joint training of the definition encoder and the scenario encoder is achieved by integrating their respective loss functions, so as to realize borrowing the large number of texts to be disambiguated provided in the training set to improve the lack of semantic knowledge caused by fixed definition descriptions in the word sense annotation texts, and then realize enhancing the representation ability of the definition embedding by borrowing the knowledge of the texts to be disambiguated.

[0080] 4. Since the text to be disambiguated containing the target word is essentially an example sentence of the word sense, it can share a pre-trained language model with the mapping model for obtaining the scenario embedding, namely Context-BERT. The advantage of such an operation is that it saves the cost of a pre-trained language model in terms of hardware resources and training costs; at the same time, the training texts for obtaining both the scenario embedding and the target word embedding are trained on the same pre-trained language model, which will enable the pre-trained language model to be more fully optimized, and thus obtain more accurate embedding representations.

[0081] 5. The dual matching mechanism recognition scheme is an effective extension of the current single matching mechanism, which fills the gap of the dual matching mechanism in the field of word sense disambiguation; at the same time, it points out the direction for the implementation of the multi-matching mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is the overall framework structure diagram of the method of the present invention, which includes two encoders implemented by a pre-trained language model, namely a word sense definition encoder and a word sense scenario encoder, and the word sense scenario encoder is simultaneously used to obtain the target word embedding and the scenario embedding.

[0083] Figure 2a and Figure 2bExperimental analysis results of this method under the constructed head sense dataset and long-tail sense dataset respectively, where Figure 2a are the experimental results under the head sense dataset, Figure 2b are the experimental results under the long-tail sense dataset.

[0084] Figure 3a and Figure 3b are the experimental analysis results of the method of the present invention under the ablation experiment, where Figure 3a are the ablation experiment analysis results of the sense definition encoder, Figure 3b are the ablation experiment analysis results of the sense scenario encoder. Detailed implementation manners

[0085] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0086] The present invention provides a long-tail sense disambiguation method integrating a definition and a scenario matching mechanism, which includes the following contents:

[0087] (1) Perform sense recognition on the target word under the definition matching mechanism:

[0088] In order to perform sense disambiguation on the target word under the definition matching mechanism, it is necessary to obtain the word embedding of the target word based on the given text to be disambiguated, obtain the definition embeddings of each sense in the sense list corresponding to the target word based on the given sense annotation text in the dictionary, and finally calculate the similarity between the target word embedding and each definition embedding to determine the score of each sense. The specific operation details are divided into the following three steps:

[0089] (101) Obtain the word embedding of the target word, that is, the target word embedding:

[0090] As Figure 1 shown, for a given text to be disambiguated text, and the text contains the target word w target to be disambiguated, then the text to be disambiguated can be formally described as

[0091] W text =[w1, w2,..., w i ,...]. (1)

[0092] It should be noted here that the number of target words to be disambiguated in the text is not limited. For the convenience of demonstration and easy understanding, this example only gives one target word to be disambiguated, that is, w target , and w target ∈W text .

[0093] First, we use the common practice of the pre-trained language model BERT to add a start marker [CLS] and a segmentation marker [SEP] at the beginning and end of the text respectively. It should be noted that the end here refers to the end of the paragraph and sentence level, and the segmentation marker [SEP] is not added at the end of the clause level. The text after adding the markers is

[0094] W text =[[CLS],w1,w2,...,w i ,...,[SEP]]. (2)

[0095] Next, we use the BERT model's mapping method, BertTokenizer(·), to encode the words in the disambiguated text, mapping each word to a specific number. This mapping is provided by the BERT model, with the start token [CLS] mapped to 101 and the segment token [SEP] mapped to 102. It's also important to note that the past tense and past participle of English verbs are still processed using the BERT model's common practice of splitting them into two items.

[0096] The encoded text to be disambiguated is called the encoded text, such as

[0097]

[0098] in

[0099] Again, the BERT model’s vectorization method Bert(·) is used to vectorize the encoded text to obtain the word embedding corresponding to each number:

[0100]

[0101] The target word is vectorized as v target When the target word to be disambiguated is the past tense and past participle of a verb, since it is split into two items during encoding, they need to be merged here. The specific operation is to add the vectors bit by bit.

[0102] Finally, the target word embedding is defined by the word embedding corresponding to the target word, i.e.

[0103] V target ≡v target (7)

[0104] (102) Obtain the definition embedding of each word in the list of word meanings corresponding to the target word:

[0105] like Figure 1 As shown, for the target word in the dictionary, the annotation text gloss of each word in the word meaning list j , which can be formally described as

[0106]

[0107] The index j represents the index of the jth word in the word sense list. It should be noted that the word sense annotation does not include the target word, that is,

[0108] First, we use the common practice of the pre-trained language model BERT to add a start marker [CLS] and a segmentation marker [SEP] at the beginning and end of the text respectively. The addition rules are the same as those described above. The text after adding the markers is

[0109]

[0110] Secondly, we use the BERT model’s mapping method BertTokenizer to encode the words. Its encoding rules are consistent with the method described above. The encoded text is

[0111]

[0112] Again, the BERT model’s vectorization method Bert(·) is used to vectorize the encoded text to obtain the word embedding corresponding to each number:

[0113]

[0114] Finally, we adopt the common practice in the industry and use the word embedding corresponding to the start tag [CLS] as the text embedding of the word meaning annotation text, that is, definition embedding.

[0115]

[0116] (103) Calculate the scores of each meaning of the target word under the definition matching mechanism:

[0117] Based on the target word embedding v obtained above target The definition embeddings of each word in the list of meanings corresponding to the target word Calculate the similarity between the target word embedding and each definition embedding to obtain the score of each word meaning, that is,

[0118]

[0119] The symbol ⊙ represents the dot product operation.

[0120] Then the score of each word meaning in the word meaning list corresponding to the target word under the definition matching mechanism is

[0121]

[0122] (2) Implement word sense recognition under the scenario matching mechanism for the target word:

[0123] To implement the word sense disambiguation strategy for the target word under the scenario matching mechanism, it is necessary to obtain the word embedding of the target word based on the given text to be disambiguated, obtain the scenario embeddings of each word sense in the word sense list corresponding to the target word based on the given word sense example text in the dictionary, and finally calculate the similarity between the target word embedding and each scenario embedding respectively to determine the scores of each word sense. The specific operation details are divided into the following three steps:

[0124] (201) Obtain the word embedding of the target word, that is, the target word embedding:

[0125] The method for obtaining the word embedding of this part of the target word is the same as that described in (101). During the implementation of the word sense disambiguation system, the word embeddings of these two target words can be shared. Experimental analysis found that the experimental results of sharing the target word embedding are better than those obtained by using different pre-trained language models for the target word embedding. The reason is that after sharing the target word embedding, the definition matching mechanism and the scenario matching mechanism can restrict each other to avoid the situation where the word embedding goes to a local optimum during the optimization process, such as being effective only for the recognition of head word senses and ineffective for the recognition of long-tail word senses.

[0126] For the convenience of reading, the formal description of the target word embedding is given here again:

[0127] V target ≡v target ∈V text (17)

[0128] where V text represents the text to be disambiguated containing the target word, and v target represents the word embedding of the target word in the text to be disambiguated.

[0129] (202) Obtain the scenario embeddings of each word sense in the word sense list corresponding to the target word:

[0130] As Figure 1 shown, for the example text example of each word sense in the word sense list corresponding to the target word in the dictionary k , it can be formally described as

[0131]

[0132] where the index k represents the index of the example sentence corresponding to the k-th word sense in the word sense list. It should be noted that the word sense example text contains the target word, that is, [[ID=*46]] <*

[0133] Since not all the meanings in the dictionary provide only one example sentence, and the proportion of those providing only one example sentence is relatively large, for the sake of convenience in processing, a single example sentence is uniformly used. For the cases where the number of example sentences for a meaning in the dictionary is greater than one, the example sentence with the largest number of text words is uniformly selected; for the cases where no example sentence is provided in the dictionary, the text to be disambiguated is uniformly adopted as the example sentence.

[0134] First, following the common practice of the pre-trained language model BERT, the start token [CLS] and the separator token [SEP] are respectively added at the beginning and the end of the text, and the addition rules are the same as those described above. Then the text after adding the tokens is

[0135]

[0136] Since the scenario information of the meaning needs to be obtained here, the target word in the text needs to be replaced with the mask token [MASK], that is

[0137]

[0138] Second, the mapping method BertTokenizer(·) of the BERT model is used to encode the words, and the encoding rules are the same as those described above. Then the encoded text is

[0139]

[0140] Third, the vectorization method Bert(·) of the BERT model is used to vectorize the encoded text to obtain the word embeddings corresponding to each number:

[0141]

[0142] Finally, following the common practice in the industry, the word embedding corresponding to the start token [CLS] is used as the scenario embedding of this meaning, that is

[0143]

[0144] (203) Calculate the scores of each meaning of the target word under the scenario matching mechanism:

[0145] Based on the obtained target word embedding v target and the scenario embeddings of each meaning in the list of meanings corresponding to the target word calculate the similarity between the target word embedding and each scenario embedding respectively to obtain the scores of each meaning,

[0146]

[0147] where the symbol ⊙ represents the dot product operation.

[0148] Then the scores of each sense of the target word under the scenario matching mechanism are

[0149]

[0150] (3) Integrate the results under the definition and scenario matching mechanisms to determine the final sense of the target word:

[0151] Such as Figure 1 As shown, the final output of this method is the probability superposition sum of the calculation results under the definition matching mechanism and the scenario matching mechanism, that is

[0152] Score = α·Score definition +β·Score context (28)

[0153] Where and α + β = 1.

[0154] In addition, it should be noted that: α and β can be set as fixed values, or can be variable values learned based on other parameters. Through experimental analysis, it is found that the best experimental effect is obtained when they are set to equal values, that is, α = β = 1 / 2. Because when they are set to equal values, the roles or contributions of the two matching mechanisms can play equal values; when choosing unequal values, artificially favoring one side is actually not conducive to balancing their respective contributions.

[0155] In addition, during the model training process, the optimizer is the mainstream learning rate adaptive optimization algorithm Adam, and its loss function is

[0156]

[0157]

[0158] where the index represents the index of each sense in the sense inventory of the target word.

[0159] Specifically, to verify the effectiveness of the method of the present invention in a real system, a sense disambiguation system is constructed based on the present invention here to verify the value and contribution of the invention in specific tasks. The specific steps are as follows:

[0160] (1) Dataset:

[0161] Standard evaluation experiment settings: This evaluation setting is in the standard English full-word sense disambiguation evaluation framework [1]The following experiments are carried out for verification. Its training set is SemCor; the development set is SemEval - 2007 (SE07); the test sets are Senserval - 2 (SE2), Senserval - 3 (SE3), SemEval - 2013 (SE13), and SemEval - 2015 (SE15). In addition, the integration of all test sets is also used as a test set (referred to as the integrated test set), and the verbs (referred to as the verb set), nouns (referred to as the noun set), adjectives (referred to as the adjective set), and adverbs (referred to as the adverb set) in it are also used as separate test sets respectively. The statistical information of each data set is shown in Table 1, including the number of documents (#Docs), the number of paragraphs (#Sents), the number of words (#tokens), the number of target words to be annotated (#Annos), the number of word senses (#Senses), the number of word senses used (#Lemmas), and the ambiguity of the data set (#Ambiguity).

[0162] Table 1: Information Statistical Table of the Development Set and Test Sets

[0163] Dataset #Docs #Sents #Tokens #Annos #Senses #Lemmas #Ambiguity SE2 3 242 5766 2282 1335 1093 5.4 SE3 3 352 5541 1850 1167 977 6.8 SE07 3 135 3201 455 375 330 8.5 SE13 13 306 8391 1644 827 751 4.9 SE15 4 138 2604 1022 659 512 5.5

[0164] Enhanced Evaluation Experiment Settings: On the basis of the original training set, the data set WNGT is incorporated to construct an enhanced training set. The reason for constructing this training set is that the current word sense disambiguation system is getting larger and larger, and more and more parameters need to be learned. Many comparison systems have adopted the method of expanding the training set to deal with this situation. Therefore, for the rigor of the comparative experiment, an enhanced training set is also provided here. In addition, the development set and test sets still use the data sets of the standard evaluation experiment settings.

[0165] Head Word Sense and Long - Tail Word Sense Evaluation Settings: Since the outstanding contribution of this invention lies in long - tail word sense disambiguation, a head word sense test set and a long - tail word sense test set are constructed here. The construction method is to classify the head word senses in the integrated test set of the test sets into the head word sense test set, and the long - tail word senses into the long - tail word sense test set. In addition, the training set and the development set still use the data sets of the standard evaluation experiment settings.

[0166] Finally, all the results in the experiment use the percentage of the general F1 - score as the evaluation index.

[0167] (2) Comparison Systems:

[0168] The word sense disambiguation system constructed by the method of the present invention is compared with current advanced and representative systems to clarify the effectiveness of the method of the present invention. The comparison systems include LMMS[2], EWISE[3], GlossBERT[4], SREF[5], ARES[6], EWISER[7], BEM[8], SparseLMMS[9], COF

[10] , ESR

[11] , Multi-Label

[12] , SACE

[13] .

[0169] LMMS: This work proves that context embedding can be used in the word sense disambiguation task and can effectively promote the final performance of the system.

[0170] EWISE: This work proposes a supervised model that can integrate word sense embeddings. This model replaces the original discrete label space with a continuous word sense embedding space, and the experimental results show that this method has strong generalization ability for unknown word senses.

[0171] GlossBERT: This work first proposes a solution to integrate word sense annotations for word sense disambiguation. Its core idea is to pair the context of the target word with word sense annotations for training, and finally achieve word sense disambiguation through a classifier.

[0172] SREF: This work proposes a framework to enhance synonym relationships, which uses word sense relationships to enhance the representational ability of word sense embeddings.

[0173] ARES: This work proposes a semi-supervised method for generating context-aware word embeddings for words in a lexical knowledge base. Its advantage is that the generated vectors are in the same semantic space as the vectors in the knowledge base.

[0174] EWISER: This work proposes a neural supervised architecture that can integrate graph knowledge and uses pre-trained synonym embeddings to enable the network to predict synonyms.

[0175] BEM: This work proposes to use the method of joint training of dual encoders to enhance the word sense representation using the semantic knowledge of training texts, and proves that long-tail word senses can be more effectively disambiguated by modeling their definitions.

[0176] SparseLMMS: This work introduces a method for representing synsets inspired by information theory, which establishes relationships between word senses from the perspectives of co-occurrence of word senses, word forms, etc., and fully explores the internal connections between synonyms.

[0177] COF: This work studies the contributions of lexical-level and word sense-level global contexts to word sense disambiguation, and points out that context-oriented embeddings can effectively improve the performance of word sense disambiguation systems.

[0178] ESR: This work fully integrates the semantic annotations of synonyms, example phrases or sentences, and hypernyms to enhance the representation ability of word meanings.

[0179] Multi-Label: This work regards word sense disambiguation as a multi-label classification problem, which assigns multiple word senses to each target word to achieve the purpose of integrating the structured knowledge in the semantic network.

[0180] SACE: This work enhances the learning ability of word meaning embeddings through similar word meaning contexts, not only effectively avoiding the singularity of single semantic annotation in learning word meaning embeddings, but also converting the original decision-making behavior into an interrelated decision-making process.

[0181] (3) Experimental Results and Analysis:

[0182] The experimental results under the standard evaluation experimental settings are shown in Table 2. Among them, Bi-MWSD represents the results obtained by the method of the present invention; from the experimental results, the word sense disambiguation system constructed by the present invention has achieved good performance on multiple evaluation data sets, indicating that the present invention does have a certain promoting effect on the overall performance of the system. The excellent performance on the development set indicates that the system can effectively learn the sample features that appear in the training process; at the same time, the results on the integrated set of the test set show that the system has good generalization ability on unknown data. However, whether the overall performance of the system is due to the effective recognition of long-tail word senses will be analyzed in the evaluation experiments of the long-tail word sense data set and the model ablation study.

[0183] For the poor performance on the test sets SE15, adjective set, and adverb set, the following analysis is made:

[0184] 1) By analyzing the distribution of target words in the test set SE15, it is found that the proportion of adjectives and adverbs in this data set is more than that of other test sets, and the number of word senses of the above word classes is relatively small. At the same time, from the difficulty indicators of each data set in Table 1, the difficulty of this test set is 5.5, which is relatively low. This shows that for a data set with a small number of long-tail word senses, the word sense disambiguation system constructed by the present invention cannot obtain the best performance, and conversely, it shows that when the proportion of long-tail word senses is low, the scene matching mechanism designed for long-tail word senses will drag down the overall performance of the system.

[0185] 2) Through the statistical analysis of the number of word senses of target words in the adjective set, it is found that the number of word senses in this data set is relatively small compared with other data sets, indicating that the disambiguation difficulty of this test set is relatively simple. The experimental results of the word sense disambiguation system constructed by the present invention show that the scene matching mechanism designed for long-tail word senses does indeed drag down the overall performance of the system on simple data sets. This conclusion is similar to the above conclusion.

[0186] 3) The number of senses of the target words concentrated by adverbs is also small, and this result also illustrates the above conclusion.

[0187] Table 2: Experimental results under the standard evaluation settings

[0188]

[0189] The experimental results under the enhanced evaluation experimental settings are shown in Table 3. From the experimental results, it can be seen that training the model on the enhanced training set by the sense disambiguation system constructed by the present invention can indeed improve the overall performance of the system. However, the system still does not perform optimally on the test sets SE15, adjective set, and adverb set. The reason is still that the scenario matching mechanism designed for long-tail senses drags down the overall performance of the system. By adjusting the proportion of the contribution of the definition matching mechanism and the scenario matching mechanism, this phenomenon will improve, but it cannot substantially change this phenomenon. The experimental results on the development set are 0.1 percentage points different from those of the strongest system. Such results do not indicate that the sense disambiguation system constructed by the present invention is relatively weak. The performance on the test set SE2 is weaker than that of the strongest system, but by comparing the performance of other systems on this data set, the sense disambiguation system constructed by the present invention still has a certain competitiveness.

[0190] Table 3: Experimental results under the enhanced evaluation settings

[0191]

[0192] In order to further clarify the specific value of each part of the present invention for head senses and long-tail senses, a test set for head senses and long-tail senses is specifically designed here. The experimental results on the head sense test set are as Figure 2a shown. The comparison systems selected are the two systems most similar to the present invention, namely GlossBERT and BEM. From the experimental results, it can be seen that the overall performance of the sense disambiguation system constructed by the present invention is better than that of GlossBERT, but inferior to BEM, indicating that for head senses, this system does not have an advantage. The possible reason is still that the scenario matching mechanism designed for long-tail senses drags down the recognition ability of head senses to a certain extent. The experimental results on the long-tail sense test set are as Figure 2b shown. The comparison systems still use GlossBERT and BEM. From the experimental results, it can be seen that the sense disambiguation system constructed by the present invention comprehensively surpasses the comparison systems, indicating that the scenario matching mechanism for long-tail senses proposed by the present invention makes an outstanding contribution to the recognition of long-tail senses. From this point of view, it is necessary to add a scenario matching mechanism for the recognition of long-tail senses to the system, and it can effectively improve the recognition ability of long-tail senses.

[0193] (4) Model ablation study:

[0194] Dividing the head sense and long-tail sense test sets has been able to prove the effectiveness of the scene matching mechanism. However, for the sake of rigorous analysis, the traditional method of model ablation study is adopted here to further study the contribution of each component of the model to the overall performance of the system from the perspective of the model.

[0195] For the Definition-BERT, the methods of freezing the model parameters (the corresponding ablation model is abbreviated as frozen) and directly removing this part (the corresponding ablation model is abbreviated as removed) are used to analyze the role of the Definition-BERT in the overall performance of the system. Freezing the model parameters means preventing the pre-trained language model from updating parameters during training, and directly removing this part means not using the definition matching mechanism. The experimental results are as Figure 3a shown. From the experimental results, the performance of the original system (the corresponding model is abbreviated as original) is much better than that of the other two ablation models, and the method of freezing the model parameters is better than the method of directly removing this part. This shows that the definition matching mechanism is useful, and it also shows that even if the pre-trained language model does not learn the feature information in the training set, it still has a certain role.

[0196] For the Context-BERT, the methods of freezing the model parameters and directly removing this part are also adopted. It should be emphasized here that directly removing this part means removing the function of the scene matching mechanism, but still retaining the learning ability of the target word embedding. In addition, since this pre-trained language model is responsible for learning both the target word embedding and the scene embedding at the same time, an ablation method of separating functions (the corresponding model is abbreviated as split) is added to the analysis of this pre-trained language model. Separating functions means using two pre-trained language models to respectively implement the functions originally implemented by one pre-trained language model. The experimental results are as Figure 3b shown. From the experimental results, the original model performs the best, the ablation model with frozen model parameters performs the worst, the ablation models of directly removing this part and separating functions are slightly lower than the original model, and the ablation model of directly removing the scene matching mechanism is better than the ablation model of separating functions. The reason why the ablation model with frozen model parameters performs the worst is that it cannot learn effective target word embeddings, so the performance is poor. The reason why the ablation models of directly removing this part and separating functions are slightly lower than the original model is that the proportion of long-tail senses in the integration set of the test set is not large, and these operations will not have much impact on the overall performance of the system. The reason why the ablation model of directly removing the scene matching mechanism is better than the ablation model of separating functions is that the scene matching mechanism cannot obtain the knowledge provided during the training process of the target word embedding, so it cannot effectively learn the scene embedding.

[0197] Finally, it should be pointed out that the above examples are only used to illustrate the calculation process of the present invention, rather than limiting it. Although the present invention has been described in detail with reference to the foregoing examples, those of ordinary skill in the art should understand that they can still modify the calculation process recorded in the foregoing examples, or make equivalent substitutions for some of the parameters, and these modifications or substitutions will not make the essence of the corresponding calculation method deviate from the spirit and scope of the calculation method of the present invention.

[0198] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the spirit and scope of the present invention as defined by the claims, those of ordinary skill in the art can make many specific variations in form under the inspiration of the present invention, and these all fall within the protection scope of the present invention.

[0199] References:

[0200] [1] Alessandro Raganato, José Camacho-Collados, Roberto Navigli: Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison. EACL 2017: 99 - 110

[0201] [2] Daniel Loureiro, Alípio Jorge: Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation. ACL 2019: 5682 - 5691

[0202] [3] Sawan Kumar, Sharmistha Jat, Karan Saxena, Partha P. Talukdar: Zero-shot Word Sense Disambiguation using Sense Definition Embeddings. ACL 2019: 5670 - 5681

[0203] [4]Luyao Huang,Chi Sun,Xipeng Qiu,Xuanjing Huang:GlossBERT:BERT forWord Sense Disambiguation with Gloss Knowledge.EMNLP / IJCNLP 2019:3507-3512

[0204] [5]Ming Wang,Yinglin Wang:A Synset Relation-enhanced Framework with aTry-again Mechanism for Word Sense Disambiguation.EMNLP 2020:6229-6240

[0205] [6]Bianca Scarlini,Tommaso Pasini,Roberto Navigli:With More ContextsComes Better Performance:Contextualized Sense Embeddings for All-Round WordSense Disambiguation.EMNLP 2020:3528-3539

[0206] [7]Michele Bevilacqua,Roberto Navigli:Breaking Through the 80%GlassCeiling:Raising the State of the Art in Word Sense Disambiguation byIncorporating Knowledge Graph Information.ACL 2020:2854-2864

[0207] [8]Terra Blevins,Luke Zettlemoyer:Moving Down the Long Tail of WordSense Disambiguation with Gloss Informed Bi-encoders.ACL 2020:1006-1017

[0208] [9] Gábor Berend: Sparsity Makes Sense: Word Sense Disambiguation Using Sparse Contextualized Word Representations. EMNLP 2020: 8498 - 8508

[0209]

[10] Ming Wang, Jianzhang Zhang, Yinglin Wang: Enhancing the Context Representation in Similarity-based Word Sense Disambiguation. EMNLP 2021: 8965 - 8973

[0210]

[11] Yang Song, Xin Cai Ong, Hwee Tou Ng, Qian Lin: Improved Word Sense Disambiguation with Enhanced Sense Representations. EMNLP (Findings) 2021: 4311 - 4320

[0211]

[12] Simone Conia, Roberto Navigli: Framing Word Sense Disambiguation as a Multi-Label Problem for Model-Agnostic Knowledge Integration. EACL 2021: 3269 - 3275

[0212]

[13] Ming Wang, Yinglin Wang: Word Sense Disambiguation: Towards Interactive Context Exploitation from Both Word and Sense Perspectives. ACL / IJCNLP 2021: 5218 - 5229

Claims

1. A long-tail word sense disambiguation method integrating a definition and a scenario matching mechanism, characterized in that Including: Text embedding for learning word meaning definitions from the word meaning annotation text in the dictionary, that is, definition embedding. The text-to-vector mapping model uses the pre-trained language model BERT and is called the definition encoder Definition-BERT; Text embedding for learning word meaning scenarios from the word meaning example text in the dictionary, that is, scenario embedding. The text-to-vector mapping model uses the pre-trained language model BERT and is called the scenario encoder Context-BERT; During the process of obtaining scenario embedding, the target word contained in the input text is hidden to obtain the scenario information of the word meaning; Word embedding of the target word is learned from the text to be disambiguated containing the target word, that is, target word embedding. The text-to-vector mapping model uses the pre-trained language model BERT; the pre-trained language model for obtaining target word embedding shares a BERT model with the pre-trained language model for obtaining scenario embedding; Based on the target word embedding and the definition embeddings of each meaning in the meaning list corresponding to the target word in the dictionary, the similarity between the target word embedding and each definition embedding is calculated respectively to obtain the scores of each meaning in the meaning list under the definition matching mechanism; Based on the target word embedding and the scenario embeddings corresponding to each meaning in the meaning list corresponding to the target word, the similarity between the target word embedding and each scenario embedding is calculated respectively to obtain the scores of each meaning in the meaning list under the scenario matching mechanism; The probability superposition sum of the scores calculated under the definition matching mechanism and the scenario matching mechanism is used as the final output value to determine the final meaning of the target word.

2. The long-tail word sense disambiguation method integrating a definition and a scenario matching mechanism according to claim 1, characterized in that Specifically, obtaining definition embedding includes: (101)Encoding the word meaning annotation text: Add the start marker [CLS] and the separation marker [SEP] at the beginning and end of the word meaning annotation text respectively, and then use the mapping method BertTokenizer(·) of the BERT model to encode the words in the word meaning annotation text with added markers into corresponding numbers to obtain the encoded text of the word meaning annotation text; (102)Vectorizing the encoded text of the word meaning annotation text to obtain definition embedding: Obtain the word embeddings corresponding to the numbers in the encoded text through the vectorizing method Bert(·) of the BERT model, that is, the word embeddings corresponding to the words in the word meaning annotation text.

3. The method for long-tail word sense disambiguation integrating a definition and a scenario matching mechanism according to claim 2, characterized in that, For the semantic annotation text W gloss , W gloss = [w1, w2, …, w i , …] (1) where w i denotes a word constituting a sense annotation text; each w in the formula i omits the subscript gloss; The form after adding the start and separation markers is: W gloss = [[CLS], w1, w2, …, w i , …, [SEP]] (2) The mapping method using the BERT model, BertTokenizer(· encodes the words in the sense annotation text into corresponding numbers to obtain the encoded text C of the sense annotation text gloss , C gloss = BertTokenizer(W gloss ) (3) Obtain the encoded text C through the vectorization method Bert(·) of the BERT model gloss The word embeddings corresponding to the numbers in Among them and respectively represent the word embeddings corresponding to the start and end markers in the encoded text; The corresponding word embedding will be used as the text embedding of the sense annotation text, that is, the definition embedding.

4. A long-tail word sense disambiguation method integrating a definition and a scenario matching mechanism according to claim 1, characterized in that Specifically, obtaining scenario embedding includes: (201)Encoding the word meaning example text: Add the start marker [CLS] and the separation marker [SEP] at the beginning and end of the word meaning example text respectively, and the target word in the word meaning example text is replaced with the mask marker [MASK]; then use the mapping method BertTokenizer(·) of the BERT model to encode the words in the word meaning example text with added markers into corresponding numbers to obtain the encoded text of the word meaning example text; (202)Vectorizing the encoded text of the word meaning example text to obtain scenario embedding: Obtain the word embeddings corresponding to the numbers in the encoded text of the word meaning example text through the vectorizing method Bert(·) of the BERT model, that is, the word embeddings corresponding to the words in the word meaning example text.

5. The method for long-tail word sense disambiguation integrating a definition and a scenario matching mechanism according to claim 4, wherein When the example sentence text corresponding to the word meaning in the dictionary does not exist, the disambiguation text containing the target word is default selected as the example sentence text for extracting scenario information; when the number of example sentence texts corresponding to the word meaning in the dictionary is more than one, the example sentence with the largest number of words in the example sentence texts is default selected as the example sentence text for extracting scenario information; for the example sentence text W example , W example = [w1, w2, …, w target , …, w i , …] (7) where w i refers to the words that make up the text of the sense example sentence; w target represents the target word in the text of the sense example sentence; each w in the formula i omits the subscript example; The form after adding the start and segmentation markers and replacing the mask markers is as follows: W example = [[CLS], w1, w2, …, [MASK], …, w i , …, [SEP]] (8) Then, using the encoding method BertTokenizer(·) of BERT, the words in the sense example text are encoded into corresponding numbers to obtain the encoded text of the sense example text: C example = BertTokenizer(W example ) (9) Based on the encoded text of the sense example text, the word embeddings corresponding to the numbers in the encoded text are obtained through the Bert(·) method of BERT, that is, the word embeddings of the corresponding words in the sense example text: Among them and respectively represent the word embeddings corresponding to the start and end markers in the encoded text; v [MASK] represents the word embedding corresponding to the mask marker; The represented word embeddings are used as scene embeddings corresponding to the semantic meanings of the words, that is 6. The long-tail word sense disambiguation method integrating a definition and a scenario matching mechanism according to claim 1, characterized in that Obtaining the target word embeddings in the text to be disambiguated specifically includes: (301) Encode the text to be disambiguated containing the target word: For the text to be disambiguated W containing the target word text , W text = [w1, w2, …, w i , …] (13) where w i refers to the words that make up the text to be disambiguated, and each w in the formula i omits the subscript text; the target word w target belongs to W text , that is, w target ∈W text ; Then, the start marker [CLS] and the segmentation marker [SEP] are added at the beginning and end of the text to be disambiguated respectively, and it becomes: W text = [[CLS], w1, w2, …, w i , …, [SEP]] (14) Then, use the mapping method BertTokenizer(·) of the BERT model to encode the words in W text into corresponding numbers to obtain the encoded text of the text to be disambiguated, that is C text = BertTokenizer(W text ) (15) (302) Vectorize the encoded text to obtain the word embeddings of the target word in the text to be disambiguated, that is, the target word embeddings: Obtain the word embeddings corresponding to the numbers in the encoded text through the vectorization method Bert(·) of the BERT model: Among them and respectively represent the word embeddings corresponding to the start and end markers in the encoded text; v target The corresponding word embedding is the word embedding of the target word; The word embedding v corresponding to the target word here target The target word embedding V as the final output target , namely V target ≡v target (18).

7. The long-tail word sense disambiguation method integrating a definition and a scenario matching mechanism according to claim 1, characterized in that Define the probability superposition sum of the calculation results under the matching mechanism and the scenario matching mechanism as the final output value to determine the final sense of the target word, specifically including: (401) Calculate the similarity between the target word embeddings and each definition embedding in the sense list corresponding to the target word: Based on the obtained target word embedding V target and the definition embeddings of all the senses in the sense inventory corresponding to the target word where the index i refers to the i-th sense in the sense inventory of the target word, and the scores of the target word belonging to each sense under the definition matching mechanism are where the operator ⊙ represents the vector dot product operation; (402) Calculate the similarity between the target word embeddings and each scenario embedding in the sense list corresponding to the target word: Based on the target word embedding V target and the scene embeddings of each sense in the sense list corresponding to the target word where the index i refers to the i-th sense in the sense list, and the scores of the target word belonging to each sense under the scene matching mechanism are (403) Perform probabilistic summation on the scores of each word meaning obtained under the definition matching mechanism and scenario matching, and use it as the final output value Score i , where the word meaning corresponding to the maximum output value is the most likely word meaning of the target word in the text to be disambiguated: wherein and α + β = 1; α and β are set as fixed values or variables learned based on other parameters; In addition, during the model training process, the optimizer is the mainstream learning rate adaptive optimization algorithm Adam, and its loss function is where the index index represents the index of the sense in the sense list corresponding to the target word.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the long-tail sense disambiguation method for fusing the definition and scenario matching mechanism according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the long-tail sense disambiguation method for fusing the definition and scenario matching mechanism according to any one of claims 1 to 7.

Images