A Long-Tail Word Sense Disambiguation Method Incorporating Decoupled Representations

Abstract

The present invention discloses a long-tail word sense disambiguation method integrating decoupled representation, including: learning the word embedding of the target word from the text to be disambiguated, that is, the target word embedding, where the text-to-vector mapping model is implemented by a target word encoder; learning the text embedding of the word sense definition from the word sense definition text in the dictionary, that is, the word sense definition embedding, where the text-to-vector mapping model is implemented by a definition encoder; duplicating the obtained target word embedding and word sense definition embedding, one directly used to calculate the similarity score of the word sense under the traditional representation method; the other is reshaped by the decoupled representation method to obtain the similarity score of the word sense under the decoupled representation, and finally the scores under the two representation methods are weighted and summed as the output value; the decoupled representation method is a representation method inspired by the entangled state in quantum theory, based on the VAE model framework, and can effectively reduce the sampling noise of the original VAE model.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and machine learning, and specifically to a long-tail word sense disambiguation method integrating decoupled representations. Background Art

[0002] With the continuous development and improvement of storage devices, computing resources, and artificial intelligence technologies, the practicality and usability of intelligent systems have been gradually improved and widely applied in all aspects of work, study, and life. For example, intelligent products based on face recognition technology are applied in many fields such as airport security checks, mobile banking, and digital marketing to achieve accurate user identification; intelligent products based on speech recognition technology are integrated into many scenarios such as in-vehicle navigation, smart homes, and wearable devices to obtain a more convenient interaction experience. It can be said that intelligent systems are gradually integrating into all fields of our lives and will profoundly affect our lifestyle.

[0003] Word Sense Disambiguation (WSD) aims to determine the most likely sense in a list of senses for a target word based on the given context, where the target word refers to the word to be disambiguated, and the list of senses refers to the set of all sense definitions (or annotations) of the target word listed in the dictionary. Word Sense Disambiguation belongs to the most fundamental research topic in the field of Natural Language Processing (NLP), belongs to the research task at the lexical level, and is a standard classification problem. However, the importance of Word Sense Disambiguation is self-evident, and the accuracy of sense recognition will directly affect the final performance of downstream tasks, such as natural language understanding, natural language generation, and human-computer dialogue tasks.

[0004] Currently, word sense disambiguation systems have been able to handle the recognition tasks of high-frequency (head) senses well, that is, for the common senses of target words, word sense disambiguation systems can correctly recognize them. However, for low-frequency (long-tail) senses, current word sense disambiguation systems are not yet able to effectively handle them. The reasons for the difficulty in long-tail word sense disambiguation are: (1) insufficient training samples for long-tail senses; (2) lack of clear and distinguishable definition descriptions for long-tail senses. It is precisely because long-tail senses lack clear definitions that ambiguity will occur during use, so people will try to avoid them as much as possible in daily life, which in turn leads to a decrease in their occurrence frequency. In fact, the concept of "word sense" itself is controversial, and humans do not always reach a consistent definition description for all word senses.

[0005] The main ideas of researchers to address long-tail word sense disambiguation can be summarized into methods of data augmentation, knowledge transfer, and imposing external constraints. The method of data augmentation attempts to construct or introduce external data to improve the disadvantaged position of long-tail word senses during model training. The most representative methods among them are integrating example sentences in the dictionary to increase the number of training samples, or introducing multilingual word sense definition texts to improve the accuracy of word sense definition representation. The method of knowledge transfer attempts to expand the context of the target word to enhance the semantic information of long-tail word senses, or fuse semantic knowledge from other domains to strengthen the representation effect of long-tail word sense definitions. The most concerned method among them is to adopt the joint training method of dual encoders (i.e., the target word encoder and the word sense definition encoder) to use the semantic knowledge in the training samples to strengthen the representation of long-tail word sense definitions. The advantage of this method is that it does not introduce external knowledge, but transfers the semantic information in the training samples to enhance the representation of word sense definitions. The method of imposing external constraints focuses on the internal positional relationship between word sense representations, attempts to map word sense representations to a continuous semantic space to achieve mutual constraints between representations, and ultimately aims to calibrate long-tail word sense representations with head word sense representations. The most effective method among them is to map the word sense definition representations in the original discrete space to a continuous space to obtain effective inferences for long-tail word sense definitions or unknown word sense definitions.

[0006] Disentangled Representation (DR), also known as disentangled representation, is to map the entangled feature representations in the original feature space to a feature-independent representation space. In this space, there is no correlation constraint between features, that is, the features are separated from each other. For example, when the feature vector of a face image is mapped to a latent representation feature space, features such as hair color, face shape, and skin color of the face are separated from each other in this space. The feature attributes of disentangled representation usually have interpretable semantic information and can be specifically operated to achieve targeted regulation of the specific attributes of the target object. For example, changing the skin color and face shape of a face, adding glasses, etc. For the long-tail word sense disambiguation task, since obtaining disentangled representation does not require the deep neural network model to deeply extract and fuse features, it can reduce the dependence on training samples; at the same time, based on disentangled representation, the long-tail word sense disambiguation system can specifically select some decisive features to achieve the purpose of dimensionality reduction of representations. Summary of the Invention

[0007] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a long-tail word sense disambiguation method integrating disentangled representation.

[0008] This method uses a Target Word Encoder to obtain the word embedding of the target word in the text to be disambiguated, that is, the target word embedding; and uses a Definition Encoder to obtain the text embedding of the sense definition text in the dictionary, that is, the sense definition embedding. The target word embedding and the sense definition embedding are each duplicated. One copy is directly used to calculate the scores of the target word belonging to each sense definition, that is, to calculate the similarity between the target word embedding and each sense definition embedding. This score is called the sense similarity score under the traditional representation method; the other copy is reshaped into a decoupled representation using the decoupled representation method, and the similarity scores between the target word and its respective senses under the decoupled representation are calculated using the same method. This score is called the sense similarity score under the decoupled representation method. Finally, the sense similarity scores obtained under the two representation methods are weighted and summed, and used as the final sense similarity score to determine the sense to which the target word belongs.

[0009] Among them, the method for obtaining the decoupled representation is derived from the entangled state in quantum theory, and is specifically implemented as a constraint mechanism inspired by the quantum entangled state. During the system implementation process, this constraint mechanism will be added as a constraint term to the loss function of the model to constrain the target word embedding and the sense definition embedding, so as to achieve the purpose of obtaining the decoupled representation. In addition, during the training process of the model, the sense similarity score finally output by the system is directly iteratively optimized to jointly train the target word encoder and the definition encoder, so as to enhance the representation ability of the high-frequency sense embedding in the text to be disambiguated for the sense definition embedding.

[0010] The object of the present invention is achieved by the following technical solutions:

[0011] A long-tail sense disambiguation method integrating decoupled representation, comprising:

[0012] Learning the word embedding of the target word from the text to be disambiguated, that is, the target word embedding. In principle, the number of target words contained in the text to be disambiguated is more than one, but for the convenience of readers' understanding, it is assumed here that each piece of text to be disambiguated contains only one target word to be disambiguated. The encoder for obtaining the target word embedding is implemented by the pre-trained language model BERT, and this encoder is called the Target Word Encoder.

[0013] Learning the text embedding of the sense definition from the sense definition text in the dictionary, that is, the sense definition embedding. In principle, the target word corresponding to the sense will not appear in the sense definition text. The encoder for obtaining the sense definition embedding is implemented by the pre-trained language model BERT, and this encoder is called the Definition Encoder.

[0014] Duplicate the target word embedding and the semantic definition embedding obtained by the target word encoder and the definition encoder respectively. One copy is directly used to calculate the scores of the target word belonging to each semantic definition, that is, the similarity between the target word embedding and each semantic definition. This score is called the semantic similarity score under the traditional representation method. The other copy is reshaped into a decoupled representation using the decoupled representation method, and the same method is used to calculate the semantic similarity scores of the target word belonging to each semantic definition under the decoupled representation. This score is called the semantic similarity score under the decoupled representation method. Weighted sum the semantic similarity scores obtained under the two representation methods and use it as the final semantic similarity score to determine the semantic meaning to which the target word belongs.

[0015] The method for obtaining the decoupled representation is specifically a constraint mechanism inspired by the entangled state in quantum theory, which will be added as a constraint term to the loss function of the model during implementation to guide and constrain the generation of the decoupled target word embedding and the decoupled semantic definition embedding.

[0016] During the model training process, jointly train the target word encoder and the definition encoder by directly optimizing the semantic similarity score finally output by the system, so as to enhance the representation ability of the high-frequency semantic embedding in the text to be disambiguated for the semantic definition embedding.

[0017] Furthermore, the steps for obtaining the target word embedding, that is, the working principle of the target word encoder, specifically include:

[0018] (101) Encode the text to be disambiguated: For the text to be disambiguated W text ,

[0019] W text = [w1, w2,..., w i ,...] (1) where w i refers to the words that make up the text to be disambiguated (the subscript text is deliberately omitted for clarity of expression). In addition, the target word w target belongs to W text , that is, w target ∈ W text . Adopting the common practice of the pre-trained language model BERT, add the start token [CLS] and the end token [SEP] at the beginning and end of the text respectively,

[0020] W text = [[CLS], w1, w2,..., wi,..,[SEP]]; (2)

[0021] At the same time, use the encoding method BertTokenizer(·) of the BERT model to encode the words in the text with added tokens into corresponding numbers to obtain the encoded text of the text to be disambiguated.

[0022] Ct ext = BertTokenizer(W text ) (3)

[0023] Among them, the start token is encoded as the number 101, the end token is encoded as the number 102, and other words are encoded as a specific number. In addition, it should be noted that for the past tense and past participle forms of English verbs, they will be processed as two items according to the common practice of the BERT model.

[0024] (102) Vectorize the encoded text to obtain the target word embedding: Use the vectorization method Bert(·) of the BERT model to vectorize the words corresponding to the numbers in the encoded text C text into word embeddings,

[0025]

[0026] where and represent the word embeddings corresponding to the start and end tokens in the encoded text respectively. The word embedding corresponding to v target is the word embedding of the target word, that is, the target word embedding.

[0027] Finally, the target word embedding obtained after the text to be disambiguated is processed by the target word encoder is represented as:

[0028] V target ≡ v target . (6)

[0029] Furthermore, the steps to obtain the sense definition embedding, that is, the working principle of the definition encoder, specifically include:

[0030] (201) Encode the sense definition text: For the sense definition text W gloss in the dictionary,

[0031] W gloss = [w1, w2,..., w i ,... ] (7)

[0032] where w i refers to the words that make up the sense definition text (the subscript gloss is deliberately omitted for clarity of expression). Similarly, following the common practice of the BERT model, add the start token [CLS] and the end token [SEP] at the beginning and end of the text respectively,

[0033] W gloss = [[CLS], w1, w2,..., w i ,.., [SEP]]; (8)

[0034] In addition, the encoding method BertTokenizer(·) of the BERT model is used to encode the words in the text after adding tags into corresponding numbers to obtain the encoded text of the word meaning definition text,

[0035] C gloss = BertTokenizer(W gloss ). (9)

[0036] (202) Vectorize the encoded text to obtain the word meaning definition embedding: The vectorization method Bert(·) of the BERT model is used to vectorize the words corresponding to the numbers in the encoded text C gloss into word embeddings,

[0037]

[0038] where and respectively represent the word embeddings corresponding to the start and end tags in the encoded text.

[0039] Finally, following the common practice of the BERT model, the word embedding corresponding to the start tag is used as the text embedding of the text, and the word meaning definition embedding obtained after the word meaning definition text is processed by the definition encoder is represented as:

[0040]

[0041] Furthermore, the steps to obtain the final word meaning score specifically include:

[0042] (301) Calculate the similarity score of the word meaning under the traditional representation method: Based on the obtained target word embedding V target and all the word meaning definition embeddings in the word meaning list corresponding to the target word where the index k refers to the k-th word meaning in the word meaning list, and there are a total of N word meanings, then the similarity score between the k-th word meaning and the target word under the traditional representation method is:

[0043]

[0044] where the operator ⊙ represents the vector dot product operation.

[0045] (302) Calculate the similarity score of the word meaning under the disentangled representation method: Based on the obtained target word embedding V target and all the word meaning definition embeddings in the word meaning list corresponding to the target word obtain their corresponding disentangled representations via the disentangled representation method Disentangled(·),

[0046] D-V target = Disentangled(Vtarget ), (14)

[0047]

[0048] Then, the similarity score between the k-th sense and the target word under the disentangled representation method is:

[0049]

[0050] where the operator ⊙ represents the vector dot product operation. In addition, the disentangled representation method Disentangled(·) can be implemented using classical methods such as VAE or β-VAE, etc., or the disentangled representation method proposed in the present invention, which will be elaborated in the following part.

[0051] (303) Calculate the similarity score of the final sense and determine the sense to which the target word belongs: Based on the sense similarity score obtained under the traditional representation method and the sense similarity score obtained under the disentangled representation method Then, the similarity score between the k-th sense in the sense list and the target word is

[0052]

[0053] where α and β can be set as fixed values or can be variable values learned based on other parameters. α and β can be understood as weights, and then this formula can be interpreted as a weighted summation operation.

[0054] Finally, the sense with the largest similarity score in the sense list is the most likely sense of the target word.

[0055] Furthermore, the principle of the method for obtaining the disentangled representation of the disentangled representation is:

[0056] The method Disentangled(·) for obtaining the disentangled representation is not limited to the classical VAE model and its improved version β-VAE. Based on the VAE model, inspired by the entangled state in quantum theory, the present invention proposes a new method that can effectively combat the sampling noise of the VAE model. For the long-tail sense disambiguation task, since obtaining the disentangled representation does not require the deep network model to deeply extract and fuse features, it can reduce the dependence on training samples; at the same time, based on the disentangled representation, the long-tail sense disambiguation system can selectively choose some decisive features to achieve the purpose of dimensionality reduction of the representation.

[0057] The loss function of the VAE model includes the reconstruction error and the error between the latent representation and the unit Gaussian distribution measured by the KL divergence,

[0058]

[0059] where X represents the target word embedding or the semantic definition embedding, i.e., V target or V def , represents the reconstructed embedding, Z represents the latent representation obtained by the decoupled representation method, 0 represents the zero vector, I represents the identity matrix, q(·|·) refers to the conditional probability distribution, refers to the Gaussian distribution. The role of is to constrain each feature distribution in the latent representation to a unit Gaussian distribution. In achieving the goal, each feature will be a sample under the unit Gaussian distribution, and there is serious sampling noise.

[0060] This discovery does not attempt to constrain each feature distribution of the latent variable to a unit Gaussian distribution. That is to say, on the premise of the Gaussian distribution hypothesis, the correlation between features is simply decoupled through the entanglement principle of quantum states. Let and

[0061]

[0062]

[0063] where c i represents the probability amplitude of the entangled state, and SSN(·) is the square normalization function, is used to learn the mean vector of X under the multivariate Gaussian distribution, is used to learn the i-th eigenvector of the density matrix of X under the multivariate Gaussian distribution, and and can be realized by the linear mapping function, then the loss function of the VAE model can be rewritten as

[0064] <,

[0065] where Diag(·) is used to obtain the diagonal matrix of the matrix. It can be seen from the entanglement principle of quantum states that the off-diagonal elements of the density matrix ∑ describe the correlation between features. Constraining it to the diagonal matrix Diag(∑) can decouple the features, or called decoupling.

[0066] Furthermore, the training process of the model specifically includes:

[0067] The final loss function of the model includes the loss of obtaining the decoupled representation of the target word embedding,

[0068] <,

[0069] and the loss of obtaining the decoupled representation of the semantic definition embedding,

[0070]

[0071] In addition, it also includes the cross-entropy loss of the final word sense similarity score,

[0072]

[0073] where the index represents the index variable of each word sense in the corresponding word sense list of the target word,

[0074] Score = [Score 1 , Score 2 ,..., Score k ,...]; (27)

[0075] During the model training process, the optimizer adopts the mainstream learning rate adaptive optimization algorithm Adam.

[0076] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the long-tail word sense disambiguation method for fusion decoupling representation are implemented.

[0077] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the long-tail word sense disambiguation method for fusion decoupling representation are implemented.

[0078] Compared with the prior art, the beneficial effects brought by the technical solution of the present invention are:

[0079] 1. The word sense disambiguation task needs to deal with both high-frequency word senses with more training samples and long-tail word senses with fewer training samples at the same time. Traditional representation methods can effectively handle high-frequency word sense disambiguation, but it is difficult to take into account long-tail word sense disambiguation. The word sense disambiguation system constructed by the present invention simultaneously adopts traditional representation methods and decoupled representation methods, that is, traditional representation is used to handle high-frequency word sense disambiguation with more training samples, and decoupled representation is used to handle long-tail word sense disambiguation with fewer training samples.

[0080] 2. Based on the entanglement principle of quantum states, the present invention proposes a decoupled representation method on the basis of the VAE model that can effectively alleviate sampling noise. This method is used in the word sense disambiguation system to obtain the target word embedding and word sense definition embedding under decoupled representation.

[0081] 3. In the process of model construction, two encoders are adopted to obtain the target word embedding and word sense definition embedding respectively. In the process of model training, by optimizing the final word sense similarity score, joint training of the two encoders is realized, so as to enhance the representation ability of the word sense definition embedding in the dictionary by the high-frequency word sense embedding of the training samples.

[0082] 4. The present invention proposes to use decoupled representation to address long-tail word sense disambiguation, explores new solutions and ideas for tasks with insufficient training samples, and also discovers new values and contributions of decoupled representation beyond interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is the overall framework structure diagram of the system of the present invention, where the target word encoder is used to obtain the target word embedding, and the word sense definition encoder is used to obtain the word sense definition embedding; in addition, the obtained target word embedding and word sense definition embedding are duplicated, one is directly used to calculate the word sense similarity score under the traditional representation method, and the other is reshaped by the decoupled representation method into the target word embedding and word sense definition embedding under the decoupled representation, and the word sense similarity score under the decoupled representation is calculated; finally, the word sense similarity scores obtained under the two representations are weighted and summed as the final output.

[0084] Figure 2 It is the experimental result of the word sense disambiguation model under the long-tail word sense dataset, where the experimental models include the comparative model BEM with similar structure, the ablation model and the original model; in addition, the constructed long-tail word sense datasets include the cases of 1, 3, 5, 10, and ALL sample sizes respectively.

[0085] Figure 3 It is the experimental result of the word sense disambiguation model under the cross-lingual evaluation dataset, where the experimental models include the comparative model XLMR-Base, the ablation model and the original model. DETAILED DESCRIPTION OF THE INVENTION

[0086] The present invention will be further described in detail below in conjunction with the drawings and specific examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0087] The present invention provides a long-tail word sense disambiguation method integrating decoupled representation, including the following contents:

[0088] (1) Perform word sense recognition on the target word under the traditional representation method:

[0089] To achieve word sense disambiguation of the target word under the traditional representation method, it is necessary to obtain the word embedding of the target word, that is, the target word embedding, based on the given text to be disambiguated; obtain the text embedding of each word sense definition in the word sense list corresponding to the target word based on the given word sense definition text in the dictionary, that is, the word sense definition embedding; finally, calculate the similarity between the target word embedding and the text embeddings of each word sense definition in the corresponding word sense list to determine the score of each word sense. The specific operation details are divided into the following three steps:

[0090] (101) Obtain the word embedding of the target word, that is, the target word embedding: For the text to be disambiguated W containing the target word text ,

[0091] W text = [w1, w2, ..., w i , ...] (1) where w i refers to the words that make up the text to be disambiguated (the subscript "text" is deliberately omitted for clarity of expression). In addition, the target word w target belongs to W text , that is, w target ∈W text . Following the common practice of using the pre-trained language model BERT, the start token [CLS] and the end token [SEP] are added at the beginning and end of the text respectively, obtaining the following form:

[0092] W text = [[CLS], w1, w2, ..., w i ,...[SEP]]; (2)

[0093] Using the encoding method BertTokenizer(·) of the BERT model, the words in the text after adding the tokens are encoded into corresponding numbers to obtain the encoded text of the text to be disambiguated,

[0094] C text = BertTokenizer(W text ); (3)

[0095] Using the vectorization method Bert(·) of the BERT model, the words corresponding to the numbers in the encoded text C text are vectorized into word embeddings to obtain the word embeddings of each word in the text to be disambiguated,

[0096]

[0097] where and respectively represent the word embeddings corresponding to the start and end tokens in the encoded text. The word embedding corresponding to v target is the word embedding of the target word, that is, the target word embedding.

[0098] Finally, the target word embedding obtained after the text to be disambiguated is processed by the target word encoder is represented as:

[0099] V target ≡ v target . (6)

[0100] (102) Obtain the text embedding of the semantic definition, that is, the semantic definition embedding: For the semantic definition text W gloss in the dictionary,

[0101] W gloss= [w1, w2,..., w i ,... ] (7)

[0102] where w i refers to the words that make up the gloss text (the subscript gloss is deliberately omitted for clarity of expression). Similarly, following the common practice of the BERT model, start token [CLS] and end token [SEP] are added at the beginning and end of the text respectively, obtaining the following form:

[0103] W gloss = [[CLS], w1, w2,..., wi,...,[SEP]]; (8)

[0104] Using the encoding method BertTokenizer(·) of the BERT model, the words in the text with tokens added are encoded into corresponding numbers to obtain the encoded text of the gloss text,

[0105] C gloss = BertTokenizer(W gloss ); (9)

[0106] Using the vectorization method Bert(·) of the BERT model, the words corresponding to the numbers in the encoded text C gloss are vectorized into word embeddings to obtain the word embeddings of each word in the gloss text,

[0107]

[0108] where and represent the word embeddings corresponding to the start and end tokens in the encoded text respectively.

[0109] Finally, following the common practice of the BERT model's text representation, the word embedding corresponding to the start token is used as the text embedding of the text. Then, the gloss embedding obtained after processing the gloss text by the definition encoder is represented as:

[0110]

[0111] (103) Calculate the similarity scores of each sense under the traditional representation method: Based on the obtained target word embedding V target and all the gloss embeddings in the sense list corresponding to the target word where the index k refers to the k-th sense in the sense list and there are a total of N senses. Then, the similarity score between the k-th sense and the target in the traditional representation method is:

[0112]

[0113] Among them, the operator ⊙ represents the vector dot product operation.

[0114] (2) Perform word sense recognition under the decoupled representation for the target word:

[0115] To achieve word sense disambiguation for the target word under the decoupled representation method, it is necessary to reconstruct the target word embedding under the decoupled representation method through the decoupled representation method Disentangled(·) based on the obtained target word embedding; similarly, obtain the word sense definition embedding under the decoupled representation method; finally, calculate the similarity scores of each word sense under the decoupled representation in a similar manner. The specific operation details are divided into the following three steps:

[0116] (201) Obtain the target word embedding under the decoupled representation method: Based on the target word embedding V obtained above target , obtain its corresponding target word embedding through the decoupled representation method Disentangled(·),

[0117] D-V target = Disentangled(V target ). (14)

[0118] (202) Obtain the word sense definition embedding under the decoupled representation method: Based on each word sense definition embedding in the word sense list corresponding to the target word obtained above , obtain its corresponding word sense definition embedding through the decoupled representation method Disentangled(·),

[0119]

[0120] (203) Calculate the similarity scores of each word sense under the decoupled representation method: Based on the target word embedding and the word sense definition embedding under the decoupled representation, the similarity score between the k-th word sense and the target word under this representation method can be calculated as:

[0121]

[0122] Among them, the operator ⊙ represents the vector dot product operation.

[0123] (3) Integrate the word sense scores obtained under the two representation methods to determine the final word sense of the target word:

[0124] Based on the similarity scores of the word senses obtained under the traditional representation method and the similarity scores of the word senses obtained under the decoupled representation method then the final similarity score between the k-th word sense in this word sense list and the target word is:

[0125]

[0126] Among them α and β can be set to fixed values or variable values learned based on other parameters.

[0127] The loss function of the word sense disambiguation model includes the loss of obtaining the decoupled representation of the target word embedding,

[0128]

[0129] and the loss of obtaining the decoupled representation of the word sense definition embedding.

[0130]

[0131] In addition, it also includes the cross-entropy loss of the final similarity score.

[0132]

[0133] where the index represents the index variable of each word sense in the corresponding word sense list of the target word.

[0134] Score = [Score 1 , Score 2 ,..., Score k ,...]; (23)

[0135] During the model training process, the optimizer adopts the mainstream learning rate adaptive optimization algorithm Adam.

[0136] Specifically, to verify the effectiveness of the method of the present invention in a real system, a word sense disambiguation system is constructed based on the present invention to verify the value and contribution of the invention in specific tasks. The specific steps are as follows:

[0137] (1) Dataset:

[0138] Experimental verification was carried out under the standard English full-word sense disambiguation evaluation framework [1], where the training set was SemCor; the development set was SemEval-2007 (SE07); the test sets were Senserval-2 (SE2), Senserval-3 (SE3), SemEval-2013 (SE13), and SemEval-2015 (SE15). In addition, the integration of all test sets was also used as a test set (referred to as the integrated test set of the test sets, abbreviated as ALL), and the verbs (referred to as the verb set), nouns (referred to as the noun set), adjectives (referred to as the adjective set), and adverbs (referred to as the adverb set) in it were also used as separate test sets. The statistical information of each data set is shown in Table 1, including the number of documents (#Docs), the number of paragraphs (#Sents), the number of words (#tokens), the number of target words to be annotated (#Annos), the number of sense meanings (#Senses), the number of sense meanings used (#Lemmas), and the ambiguity of the data set (#Ambiguity).

[0139] In the experiment, all results were evaluated using the percentage of the general F1 value (F1-score).

[0140] Table 1: Information Statistical Table of the Training Set, Development Set, and Test Sets

[0141] Dataset #Docs #Sents #Tokens #Annos #Senses #Lemmas #Ambiguity SE2 3 242 5766 2282 1335 1093 5.4 SE3 3 352 5541 1850 1167 977 6.8 SE07 3 135 3201 455 375 330 8.5 SE13 13 306 8391 1644 827 751 4.9 SE15 4 138 2604 1022 659 512 5.5 SemCor 352 37,176 802,443 226,036 33,362 22,436 6,8

[0142] (2) Comparative System:

[0143] To verify the effectiveness of the sense disambiguation system constructed by the method of the present invention and its position in the sense disambiguation community, two sets of comparative experiments were carried out under the standard evaluation framework here, and compared with the most representative previous work, including GLU [2], LMMS [3], SREF [4], ARES [5], SyntagRank [6], COF [7], ESR [8], SACE [9], and compared with the most comparable related models, including GlossBERT

[10] , BEM

[11] , IMS

[12] , EWISE

[13] , EWISER

[14] . At the same time, based on the framework of the IMS model, three traditional representation methods were used for experiments respectively to evaluate the superiority between the decoupled representation proposed by the present invention and the traditional representation.

[0144] GLU: This work verified that both integrating different context-based word representations and integrating word representations under multiple benchmarks have outstanding performances that exceed previous work in the sense disambiguation task.

[0145] LMMS: This work shows that the contextual representation method promotes the word sense disambiguation task, and the accuracy of the representation directly determines the final performance of the system.

[0146] SREF: This work enhances the word sense representation by using the semantic information between synonyms, and the experimental results show that there is a strong correlation between synonyms, which can promote the overall performance of the representation.

[0147] ARES: This work enhances the word sense representation by using similar words in the lexical knowledge base. Its advantage is that the semantic knowledge in the knowledge base can ensure that the generated vectors are in the same semantic space.

[0148] SyntagRank: This work uses the disambiguation of co-occurring words included in SyntagNet to enhance the representation of the target word and the word sense, and confirms that the combined information resources of vocabulary and semantics can provide rich semantic knowledge.

[0149] COF: This work analyzes the value of context information for word sense disambiguation, and points out that the contextual target word and word sense representation are conducive to promoting the overall representation of the word sense disambiguation system.

[0150] SACE: This work improves the accuracy of the word sense representation by integrating the representations of similar word senses. Its advantage is that it can avoid the one-sidedness of the representation learned from a single word sense definition.

[0151] GlossBERT: This work uses the annotations in the dictionary to learn the representation of the word sense, transforming the word sense disambiguation task into a word sense matching problem. Its contribution is to shift the research focus to the distinguishability direction of the word sense representation.

[0152] BEM: This work uses the joint training method of a dual encoder (i.e., the target word encoder and the word sense definition encoder) to achieve the purpose of enhancing the word sense representation with training text knowledge.

[0153] IMS: This work inputs the text to be disambiguated and the word sense definition as a whole into the neural network to achieve word sense recognition. Its advantage is that the model structure is simple and easy to integrate into the existing model.

[0154] EWISE: This work embeds the word sense definition into a continuous space to improve the accuracy of the word sense representation. Its advantage is that it can predict the representation of unknown word senses.

[0155] EWISER: This work uses the semantic information in the graph knowledge to achieve the prediction of synonyms by the word sense disambiguation system.

[0156] (III) Experimental Results and Analysis:

[0157] The experimental results under the English full-word sense disambiguation evaluation framework are shown in Tables 2 and 3. Table 2 shows the comparative experimental results between the sense disambiguation system constructed by the present invention and previous work, and Table 3 shows the comparative experimental results between this system and related models.

[0158] Judging from the experimental performance in Table 2, the sense disambiguation system constructed by the present invention fails to achieve excellent performance on all test sets. Its performance on test sets SE13, SE15, and the adjective set is inferior to that of model SACE. The reason for this result is that model SACE uses a chain discrimination method, that is, it sequentially identifies the target words appearing in the text to be disambiguated. Its advantage is that the recognition result of subsequent target words can correct the recognition result of previous target words. However, the evaluation method of this model does not conform to the evaluation criteria set by this sense disambiguation evaluation framework, and its result is not comparable to that of the sense disambiguation system constructed by the present invention. After excluding model SACE, the performance of our model is in an advantageous position.

[0159] Table 2: Comparative experimental results with previous work

[0160]

[0161]

[0162] Judging from the experimental performance in Table 3, the sense disambiguation system constructed by the present invention comprehensively surpasses the comparative models, indicating that the setting of using two representation methods simultaneously is feasible, and it also shows that decoupled representation can be effectively applied to the sense disambiguation task. Among them, GlossBERT and BEM also use the sense definition resources in the dictionary. The experimental results show that it is necessary for our model to add decoupled representation; BEM has a similar structure to our system, and the experimental results show that decoupled representation makes a certain contribution to long-tail sense disambiguation; IMS word2vec 、IMS context2vec 、IMS bert respectively adopt different representation methods based on the IMS framework. The experimental results show that the sense disambiguation system constructed by integrating the decoupled representation method is superior to the models based on traditional representation methods, which shows that decoupled representation has certain advantages for dealing with long-tail sense disambiguation.

[0163] Table 3: Comparative experimental results with related models

[0164]

[0165] (4) Model ablation study:

[0166] To identify the contribution of decoupled representation to the integrated representation of the system, an ablation experiment was conducted on the model. Among them, the decoupled representation part was removed from the original model to construct an ablation model OURablation ; The parameter update of the input target word embedding and the input semantic definition embedding is decoupled and frozen, constructing the frozen model OUR frozen ; In addition, the original model OUR original , and the model BEM with a similar structure are both used as comparison models.

[0167] The experimental results of the ablation models under the standard semantic disambiguation evaluation framework are shown in Table 4. From the experimental performance, the original model OUR original is slightly better than the model BEM and far better than the ablation model OUR ablation and the frozen model OUR frozen . The reason is that the structure and resources used in the model BEM are similar to ours. Our model adds decoupled representation, and the decoupled representation is beneficial to long-tail semantic disambiguation. However, the proportion of long-tail semantics is small and cannot significantly improve the final experimental performance; the frozen model OUR frozen Since the input target word embedding and the input semantic definition embedding are not updated, the disambiguation decision made under the decoupled representation will instead mislead the performance of the final system; the ablation model OUR ablation is essentially the model BEM, and this experimental performance shows that there is still room for improvement in our model.

[0168] Table 4: Experimental results of ablation models

[0169]

[0170] (5) Long-tail semantic disambiguation evaluation experiment:

[0171] To verify the value of the semantic disambiguation system constructed by the present invention in long-tail semantic disambiguation, a long semantic disambiguation evaluation test set was constructed on the basis of the standard semantic disambiguation evaluation framework for verification. Training data with sample sizes of 1, 3, 5, and 10 were selected from the integrated set ALL of the test set as long-tail semantic test sets with sample sizes of 1, 3, 5, and 10. The training set and the development set still used the original training set SemCor and the development set SE07. The experimental models selected the ablation model OUR ablation and the model BEM with a similar structure as comparison models.

[0172] The experimental results are as Figure 2 shown. From the experimental performance, in the test set with a sample size of 1, the performance of all models is comparable; in the test set with a sample size of 3, the performance of the model BEM is comparable to that of the original model OUR original , and the performance of the ablation model OUR ablation decreases significantly; in the test sets with sample sizes of 5 and 10, the original model OUR originalThe advantages of the decoupled representation begin to emerge. The results show that the decoupled representation also requires a certain amount of training samples to support the learning of effective features. When the number of training samples is too small, the decoupled representation cannot play its role.

[0173] (6) Cross-language evaluation experiment:

[0174] The cross-language dataset is the latest evaluation dataset proposed by the word sense disambiguation community. Experiments on this dataset can reflect the generalization ability of the model. The experimental model includes the ablation model OUR in the ablation study part. ablation With the original model OUR original , and the model XLMR-Base used in the paper that proposed this evaluation dataset.

[0175] The experimental results are as follows Figure 3 As shown in the experimental results, the original model OUR original Outperforms the comparison model XLMR-Base and the ablation model OUR on multiple datasets ablation The overall performance is poor. Analysis found that the original model OUR original The datasets that performed poorly were mainly non-English languages. The reason for this result is that the knowledge learned in the English context cannot be effectively transferred to other non-English languages.

[0176] References:

[0177] [1]Alessandro Raganato,José Camacho-Collados,Roberto Navigli:WordSense Disambiguation:A Unified Evaluation Framework and EmpiricalComparison.EACL 2017:99-110

[0178] [2]Christian Hadiwinoto,Hwee Tou Ng,and Wee Chung Gan:Improved WordSense Disambiguation Using Pre-Trained Contextualized WordRepresentations.EMNLP 2019:5296–5305.

[0179] [3] Daniel Loureiro and Alípio Mário Jorge: Language Modelling Makes Sense: Propagating Representations through WordNet for Full-Coverage Word Sense Disambiguation. ACL 2019: 5682–5691.

[0180] [4] Bianca Scarlini, Tommaso Pasini, and R. Navigli: With More Contexts Comes Better Performance: Contextualized Sense Embeddings for All-Round Word Sense Disambiguation. EMNLP 2020: 3528–3539.

[0181] [5] Ming Wang and Yinglin Wang: A Synset Relation-enhanced Framework with a Try-again Mechanism for Word Sense Disambiguation. EMNLP 2020: 6229–6240.

[0182] [6] Federico Scozzafava, Marco Maru, Fabrizio Brignone, Giovanni Torrisi, and R. Navigli: Personalized PageRank with Syntagmatic Information for Multilingual Word Sense Disambiguation. ACL 2020: 37–46.

[0183] [7] Ming Wang, Jianzhang Zhang, and Yinglin Wang: Enhancing the Context Representation in Similarity-based Word Sense Disambiguation. EMNLP 2021: 8965–8973.

[0184] [8] Yang Song, Xin Cai Ong, Hwee Tou Ng, and Qian Lin: Improved Word Sense Disambiguation with Enhanced Sense Representations. EMNLP 2021: 4311–4320.

[0185] [9] Ming Wang and Yinglin Wang: Word Sense Disambiguation: Towards Interactive Context Exploitation from Both Word and Sense Perspectives. ACL 2021: 5218–5229.

[0186]

[10] Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang: GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge. EMNLP 2019: 3507–3512.

[0187]

[11] Terra Blevins and Luke Zettlemoyer: Moving Down the Long Tail of Word Sense Disambiguation with Gloss Informed Bi-encoders. ACL 2020: 1006–1017.

[0188]

[12] Ignacio Iacobacci, Mohammad Taher Pilehvar, and R. Navigli: Embeddings for Word Sense Disambiguation: An Evaluation Study. ACL 2016: 897–907.

[0189]

[13] Sawan Kumar, Sharmistha Jat, Karan Saxena, and Partha Pratim Talukdar: Zero-shot Word Sense Disambiguation using Sense Definition Embeddings. ACL 2019: 5670–5681.

[0190]

[14] Michele Bevilacqua and R. Navigli: Breaking Through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information. ACL 2020: 2854–2864.

Claims

1. A long-tail word sense disambiguation method integrating decoupled representation, characterized in that, Including the following steps: Learning the word embedding of the target word from the text to be disambiguated, that is, the target word embedding. The encoder for obtaining the target word embedding is implemented by the pre-trained language model BERT, and this encoder is called the target word encoder; Learning the text embedding of the sense definition from the sense definition text in the dictionary, that is, the sense definition embedding. The encoder for obtaining the sense definition embedding is implemented by the pre-trained language model BERT, and this encoder is called the definition encoder; Duplicating the target word embedding and the sense definition embedding obtained by the target word encoder and the definition encoder respectively. One copy is directly used to calculate the scores of the target word belonging to each sense definition, that is, the similarity between the target word embedding and each sense definition. This score is called the score under the traditional representation method. The other copy is reshaped into a decoupled representation using the decoupled representation method, and the scores of the target word belonging to each sense definition under the decoupled representation are calculated using the same method. This score is called the score under the decoupled representation method; Performing weighted summation on the scores obtained under the two representation methods and using it as the final sense score to determine the sense to which the target word belongs; The method for obtaining the decoupled representation is specifically a variational autoencoder (VAE) inspired by the entangled state in quantum theory. This method is constructed on the basis of the VAE model and can effectively reduce the sampling noise of the VAE model; during implementation, it will mainly be presented as a constraint term added to the loss function, like the VAE model, to guide and constrain the generation of the decoupled target word embedding and the decoupled sense definition embedding; for the long-tail word sense disambiguation task, since obtaining the decoupled representation does not require the deep network model to deeply extract and fuse features, it can reduce the dependence on training samples; at the same time, based on the decoupled representation, the long-tail word sense disambiguation system can selectively choose some decisive features to achieve the purpose of dimensionality reduction of the representation; During the model training process, by directly optimizing the sense similarity score finally output by the system, the joint training of the target word encoder and the definition encoder is realized, and then the representation ability of the sense definition embedding in the dictionary is enhanced by the high-frequency word embedding in the text to be disambiguated; 2. The long-tail word sense disambiguation method with integrated decoupled representation according to claim 1, characterized in that Specifically, obtaining the target word embedding includes: (101) Perform encoding on the text to be disambiguated: For the text to be disambiguated W containing the target word text , W text = [w1, w2, …, w i , …] (1) where w i refers to the words that make up the text to be disambiguated. For the sake of clarity of expression, the subscript "text" is deliberately omitted. In addition, the target word w target belongs to W text , that is, w target ∈W text , following the common practice of using the pre-trained language model BERT, the start token [CLS] and the end token [SEP] are added at the beginning and end of the text respectively. W text = [[CLS], w1, w2, …, w i , …, [SEP]]; (2) At the same time, using the encoding method BertTokenizer(·) of the BERT model to encode the words in the text with added markers into corresponding numbers to obtain the encoded text of the text to be disambiguated, C text = BertTokenizer(W text ) (3) where the start marker is encoded as the number 101, the end marker is encoded as the number 102, and other words are encoded as a specific number; in addition, it should be noted that for the past tense and past participle forms of English verbs, they will be processed as two items according to the common practice of the BERT model; (102) Vectorize the encoded text to obtain target word embeddings: Use the vectorization method Bert(·) of the BERT model to vectorize the words with corresponding numbers in the encoded text C text into word embeddings, Among them and respectively represent the word embeddings corresponding to the start and end markers in the encoded text; v target The corresponding word embedding is the word embedding of the target word, that is, the target word embedding; Finally, the target word embedding obtained after the text to be disambiguated is processed by the target word encoder is represented as: V target ≡v target 。 (6) 3. The long-tail word sense disambiguation method with fused disentangled representation according to claim 1, wherein Specifically, obtaining the sense definition embedding includes: (201) Encode the semantic definition text: For the semantic definition text W in the dictionary gloss , W gloss = [w1, w2, …, w i , …] (7) where w i refers to the words that make up the text of the sense definition. For the sake of clarity in expression, the subscript gloss is deliberately omitted. Similarly, following the common practice of the BERT model, the start token [CLS] and the end token [SEP] are added at the beginning and end of the text respectively. W gloss = [[CLS], w1, w2, …, w i , …, [SEP]]; (8) In addition, using the encoding method BertTokenizer(·) of the BERT model to encode the words in the text with added markers into corresponding numbers to obtain the encoded text of the sense definition text, C gloss = BertTokenizer(W gloss ); (9) (202) Vectorize the encoded text to obtain the semantic definition embedding: Use the vectorization method Bert(·) of the BERT model to vectorize the words corresponding to the numbers in the encoded text C gloss into word embeddings, Among them and respectively represent the word embeddings corresponding to the start and end markers in the encoded text; Finally, following the common practice of the BERT model, the word embedding corresponding to the start token is used as the text embedding of the text. Then, the semantic definition embedding obtained after the semantic definition text is processed by the definition encoder is represented as:

4. The long-tail word sense disambiguation method with fused disentangled representation according to claim 1, wherein Obtaining the final semantic score specifically includes: (301) Calculate the semantic similarity score under the traditional representation method: Based on the obtained target word embedding V target and all the semantic definition embeddings in the semantic list corresponding to the target word where the index k refers to the k-th semantic in the semantic list, and there are a total of N semantics. Then, the similarity score between the k-th semantic and the target word under the traditional representation method is: where the operator ⊙ represents the vector dot product operation; (302) Calculate the word sense similarity score under the decoupled representation method: Based on the obtained target word embedding V target and all the sense definition embeddings in the sense inventory corresponding to the target word obtain their corresponding decoupled representations via the decoupled representation method Disentangled(·), D-V target = Disentangled(V target ), (14) Then, the similarity score between the k-th semantic meaning and the target word under the disentangled representation method is: where the operator ⊙ represents the vector dot product operation; in addition, the disentangled representation method Disentangled(·) can be implemented using classical methods such as VAE or β-VAE, etc., or the disentangled representation method proposed in the present invention can also be used, and this method will be elaborated in the following part; (303) Calculate the final semantic similarity score and determine the semantic meaning to which the target word belongs: the semantic similarity score obtained under the traditional representation method and the semantic similarity score obtained under the decoupled representation method Then the similarity score between the k-th semantic meaning in the semantic meaning list and the target word is Among them α and β can be set to fixed values or can be variable values learned based on other parameters; if α and β are understood as weights, then this formula can be interpreted as a weighted summation operation; Finally, the semantic meaning with the largest similarity score in the semantic meaning list is the most likely semantic meaning of the target word.

5. The long-tail word sense disambiguation method with fused disentangled representation according to claim 1, wherein Inspired by the quantum entanglement state, the method Desentangled(·) for obtaining disentangled representations based on the VAE model specifically includes: Based on the VAE model, the present invention is inspired by the entanglement state in quantum theory and proposes a new method that can effectively combat the sampling noise of the VAE model; the loss function of the VAE model includes the reconstruction error and the error between the hidden representation and the unit Gaussian distribution measured by the KL divergence. where X represents the target word embedding or the sense definition embedding, i.e., V target or V def , represents the reconstructed embedding, Z represents the latent representation obtained by the decoupled representation method, 0 represents the zero vector, I represents the identity matrix, q(·|·) refers to the conditional probability distribution, refers to the Gaussian distribution; serves to constrain each feature distribution in the latent representation to a unit Gaussian distribution; in the case of achieving the goal, each feature will be a sample under the unit Gaussian distribution, with serious sampling noise; This discovery does not attempt to constrain the feature distribution of each implicit representation to a unit Gaussian distribution. That is, on the premise of the Gaussian distribution hypothesis, the correlation between features is simply decoupled through the entanglement principle of quantum states; let and where c i represents the probability amplitude of the entangled state, and SSN(·) is the square normalization function, which is used to learn the mean vector of X under the multivariate Gaussian distribution, which is used to learn the i-th eigenvector of the density matrix of X under the multivariate Gaussian distribution, and and can be realized by a linear mapping function, then the loss function of the VAE model can be rewritten as where Diag(·) is used to obtain the diagonal matrix of the matrix; according to the entanglement principle of quantum states, the non-diagonal elements of the density matrix Σ describe the correlation relationship between features, and constraining it to the diagonal matrix Diag(Σ) can disentangle the features; The training process of the model specifically includes: the final loss function of the model includes the loss of obtaining the disentangled representation of the target word embedding. and the loss of obtaining the disentangled representation of the semantic definition embedding. In addition, it also includes the cross-entropy loss of the final semantic similarity score. where index represents the index variable of each semantic meaning in the corresponding semantic meaning list of the target word. Score = [Score 1 , Score 2 , …, Score k , …]; (27) During the model training process, the optimizer uses the mainstream learning rate adaptive optimization algorithm Adam.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the long-tail semantic disambiguation method for fusing disentangled representations according to any one of claims 1 to 5.

7. A computer-readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by the processor, it implements the steps of the long-tail semantic disambiguation method for fusing disentangled representations according to any one of claims 1 to 5.

Images