Semantic linking between words and definitions through semantic catalog alignment

Abstract

Methods and apparatuses are included that include computer code configured to cause one or more processors to generate one or more aligned lists, where the one or more aligned lists are generated using one or more semantic lists; obtain words of the context sentence; determine, using a semantic equivalence recognizer model, one or more semantic equivalence scores indicative of semantic similarity between the words of the context sentence and each of one or more associated phrase annotations of the one or more aligned lists; and predict correct meanings of the words of the context sentence based on the determined one or more semantic equivalence scores.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Application No. 17 / 508,417, filed with the U.S. Patent and Trademark Office on October 22, 2021. This U.S. application is incorporated herein by reference in its entirety.

[0002] Embodiments of the present disclosure are directed to the field of Natural Language Processing (NLP), and in particular to the field of Word Sense Disambiguation (WSD), which aims to automatically understand the exact meaning of a word based on its usage in a sentence or expression. [Background technology]

[0003] In some sense, human languages ​​are ambiguous since a word may have multiple meanings in different contexts. WSD aims to automatically identify the exact meaning of a word given its usage (usually a context sentence). Identifying the correct meaning of a word in context is essential for many downstream tasks such as machine translation, information extraction and other tasks in natural language processing.

[0004] One of the problems solved by this disclosure is the difficulty that supervised models face when trying to predict the correct meaning of rare senses of words due to limited training data on such rare senses. Since most models predict word meanings based on training with a predefined semantic inventory, rare words that do not occur or occur very infrequently are usually ignored when predicting the meaning of a word.

[0005] Many solutions involve fine-tuning language models using large amounts of text data on task-specific datasets. However, such solutions often limit the applicability of the trained models and have major problems. First, the performance of the model drops significantly when predicting rare zero-shot meanings due to insufficient samples in the training data. Another problem is that task-specific fine-tuning of models often makes them catalog-dependent, and the model can only select the best definition (form one default semantic catalog (e.g. WordNet)) and not select a more comprehensive one. Summary of the Invention [Means for solving the problem]

[0006] The present disclosure addresses one or more technical problems. To address the problem of correctly predicting the meaning of rare senses, i.e., the data sparseness problem, and to generalize models that are independent of one predefined catalog, the present disclosure proposes a gloss alignment algorithm that aligns glosses with the same meanings from different semantic catalogs to capture rich lexical knowledge. By training or fine-tuning a model using such aligned catalogs to identify semantic equivalences of a word in context with one of its annotations, the data sparseness problem and the generalization problem are addressed while improving predictions for both common and rare senses.

[0007] In an embodiment of the present disclosure, a method and apparatus for predicting word meaning is provided.

[0008] According to one aspect of the present disclosure, a method for predicting a meaning of a word includes generating one or more aligned lists, where the one or more aligned lists are generated using one or more semantic lists; obtaining words of a context sentence; determining, using a semantic equivalence recognizer model, one or more semantic equivalence scores indicating a semantic similarity between the words of the context sentence and each of one or more associated phrase annotations of the one or more aligned lists; and predicting a correct meaning of the words of the context sentence based on the determined one or more semantic equivalence scores.

[0009] According to an aspect of the present disclosure, generating one or more aligned catalogs includes collecting phrase annotations from a first semantic catalog; collecting phrase annotations from a second semantic catalog; and determining a best match between the first and second semantic catalogs, wherein determining a best match between the first and second semantic catalogs includes: for each common word between the first and second semantic catalogs, determining each phrase annotation from the first semantic catalog and one or more associated annotations from the second semantic catalog; and determining a matching function that maps each phrase annotation from the first semantic catalogue to each of the one or more associated phrase annotations from the second semantic catalogue, the matching function being configured to maximize a sum of the word-for-word similarity scores between each phrase annotation from the first semantic catalogue and each of the one or more associated phrase annotations from the second semantic catalogue.

[0010] According to aspects of the present disclosure, generating the one or more aligned catalogs further includes generating positive annotation pairs by pairing a phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog based on a determination that a word-for-word similarity score between the phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog exceeds a threshold, and generating negative annotation pairs by pairing a phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog based on a determination that a word-for-word similarity score between the phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog is less than a threshold.

[0011] According to an aspect of the present disclosure, determining a sentence-by-sentence similarity score between each phrase annotation from the first semantic catalog and each of the one or more associated phrase annotations from the second semantic catalog includes determining one or more sentence embeddings based on a pre-trained secondary model, and determining a cosine similarity between each phrase annotation from the first semantic catalog and each of the one or more associated phrase annotations from the second semantic catalog based on the one or more sentence embeddings.

[0012] According to aspects of the present disclosure, the pre-trained secondary model includes a Sentence Bidirectional Encoder Representations from Transformers (SBERT) model.

[0013] According to an aspect of the present disclosure, using a semantic equivalence recognizer model to determine one or more semantic equivalence scores indicative of semantic similarity between words of the context sentence and each of one or more associated phrase annotations of one or more aligned lists includes inputting the words of the context sentence into the semantic equivalence recognizer model, inputting the one or more aligned lists into the semantic equivalence recognizer model, identifying one or more phrase annotations from the one or more aligned lists associated with the words of the context sentence, and applying a trained phrase annotation classifier to the identified one or more phrase annotations to generate a probability score for each of the identified one or more phrase annotations.

[0014] According to aspects of the present disclosure, a trained phrase annotation classifier is trained using augmented training data, which is a combination of one or more aligned inventories and embedded training data associated with a particular semantic inventory.

[0015] According to aspects of the present disclosure, a trained phrase annotation classifier is trained using one or more aligned inventories, and the trained phrase annotation classifier is fine-tuned using embedded training data associated with a specific semantic inventory of the new domain.

[0016] According to aspects of the present disclosure, the one or more semantic inventories are lexical datasets for a language.

[0017] According to an aspect of the present disclosure, predicting the correct meaning of the words of the context sentence based on the determined one or more semantic equivalence scores includes selecting the result phrase annotation associated with the largest semantic equivalence score.

[0018] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1] FIG. 2 is a simplified block diagram of a semantic prediction model according to an embodiment of the present disclosure. [Diagram 2] 1 is a simplified diagram of generating an aligned lexical annotation table according to an embodiment of the present disclosure; [Diagram 3] 1 is a simplified diagram of a semantic prediction model according to an embodiment of the present disclosure. [Figure 4] 1 is a simplified flowchart of a semantic prediction model according to an embodiment of the present disclosure. [Diagram 5] 1 is a simplified flowchart of a semantic prediction model according to an embodiment of the present disclosure. [Figure 6] 1 is a simplified flowchart of a semantic prediction model according to an embodiment of the present disclosure. [Figure 7] 1 is a simplified flowchart of a semantic prediction model according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The proposed features described below may be used individually or in any combination in any order. Furthermore, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium.

[0021] 1 is a simplified diagram of a semantic prediction model 100 according to an embodiment. The semantic prediction model 100 can predict whether a word in a context sentence and a phrase annotation are semantically equivalent. Thus, the semantic prediction model 100 predicts the semantic meaning of a word in a context sentence.

[0022] The phrase annotation alignment of the semantic catalogs in operation 110 may involve aligning the multiple semantic catalogs to provide a best mapping or matching alignment of the phrase annotations from across the multiple semantic catalogs. To leverage lexical and contextual information from the multiple semantic catalogs, the phrase annotation alignment, i.e., alignment of the catalogs, may include a best matching function that includes a mapping of the phrase annotations of common words from one of the semantic catalogs to the phrase annotations of another of the semantic catalogs such that the sentence textual similarity of the mappings included in the matching function is maximized.

[0023] In operation 120, pairs of annotations may be generated, each of which may include a mapping of annotations of common words from one of the semantic libraries to annotations of another of the semantic libraries. In some embodiments, mappings in which the pairs of annotations may be aligned, i.e., where both annotations in the pair may have a high word-for-word similarity, may be labeled as positive pairs of annotations. In some embodiments, mappings in which the pairs of annotations may not be aligned, i.e., where both annotations in the pair may have a low word-for-word similarity, may be labeled as negative pairs of annotations. In some embodiments, only pairs with a word-for-word similarity above a threshold may be considered to improve the quality of the teacher and training. In some embodiments, pairs of annotations may be generated using annotations in each semantic library separately. Thus, in some embodiments, for each word in the semantic catalog, a pair of annotated sentences may be generated using example sentences to obtain positive pairs of annotations. Similarly, in some embodiments, for each word in the semantic catalog, a pair of annotated sentences may be generated using example sentences for other, unrelated words to generate negative pairs of annotations.

[0024] At 140, a context sentence containing the word for which the meaning is to be determined can be obtained. At 130 and 135, a transformer can be used to train the model using the training data. In some embodiments, the transformer can be pre-trained and applied to the context sentence to generate probabilities. At 160, the generated probabilities can be used to predict the correct meaning of the word in the context sentence.

[0025] As an example, consider an evaluation of the word semantic prediction model 100 using two WSD datasets: one focused on all-words WSD evaluation, and the other, Few Shot Examples of Word Senses (FEWS), which focuses on low-shot evaluation to understand the performance of the word semantic prediction model 100 on general semantic catalogs in the presence of data sparseness issues.

[0026] The all-words WSD and FEWS datasets are annotated using WordNet 3.0. The application may generate positive and negative pairs of word annotations generated from embedded training data from the particular dataset used for training. The application may also generate aligned catalogs using one or more dictionaries that provide rich vocabulary knowledge to the annotations. Generating aligned catalogs may include generating positive and negative pairs of word annotations from one or more dictionaries.

[0027] In some embodiments, the Transformers (130, 135) of the semantic prediction model 100 can be trained using augmented training data that combines annotation pairs from the aligned catalog and annotation pairs from embedded training data from a particular dataset. The augmented model (SemEq-Base) can be trained using only the augmented training data.

[0028] In some embodiments, the transformers (130, 135) of the semantic prediction model 100 can be first trained using training data that includes only pairs of annotations from the aligned catalogue. Using only pairs of annotations from the aligned catalogue, a general model (SemEq-Large-General) can be generated that can determine whether the words in the context sentence and the annotations are semantically equivalent or unrelated to any particular semantic catalogue. In some embodiments, the general model is further trained or fine-tuned with built-in training data for a particular semantic catalogue to create an expert model (SemEq-Large-Expert). The expert model can adapt well to new domains and achieve superior performance.

[0029] [Table 1]

[0030] As shown in Table 1, the expert model (SemEq-Large-Expert) (line 16) consistently outperforms the previous best model without WordNet synset graph information, AdaptBERT (line 9), in SE07, SE2, SE3, and SE13, achieving 1.2% higher F1 “overall”. The expert model (SemEq-Large-Expert) also disambiguates all kinds of words, including nouns, verbs, adjectives, and adverbs, better than AdaptBERT, demonstrating the effectiveness of leveraging multiple semantic inventories with lexical annotation alignment and transfer learning. The expert model (SemEq-Large-Expert) is only 0.6% more accurate when compared to EWISER (line 10), which also uses special WordNet graph knowledge. Therefore, by being pre-trained with lexical knowledge derived from the aligned catalog, the semantic prediction model can more easily generalize and better capture semantic equivalence between the words in the context description and the lexical annotations to identify the correct meaning of a word.

[0031] [Table 2]

[0032] Table 2 shows the results on the FEWS dataset. BEMSemCor (line 4) is a similar transfer learning model, but fine-tuned with SemCor before training with FEWS, whereas BEM (line 3) is trained only on FEWS. The second column shows that augmenting the FEWS training set with multiple semantic inventories using lexical annotation alignment (line 6) significantly improves the performance of zero-shot learning by 1.6% on the dev set and 2.4% on the test set (compare line 5). When transfer learning methods are employed on the FEWS dataset, the performance of the final SemEq-Large-Expert (line 10) model on the test set improves to 82.3% for few-shot semantics and 72.2% for zero-shot semantics, which is significantly better than all baseline models.

[0033] 2 is a simplified schematic diagram of an aligned semantic catalog 200. An aligned semantic catalog (also simply referred to as an aligned catalog) can include best mapping or best matching alignments of phrase annotations (210, 211, 212) from across multiple semantic catalogs (204-209).

[0034] The semantic enumerations (204-209) may be dictionaries that provide multiple example sentences for each sense of a word according to its usage, and can be used as a means of receiving context sentences for a particular sense of a word. For example, dictionaries such as Collins and Webster's may be used to provide a large database of lexical knowledge of the English language. Each of the semantic enumerations (204-209) may have multiple examples or annotations of a particular word in a limited number of contexts. Thus, annotations of a word's sense from different semantic enumerations (204-209) may be different expressions of the same meaning. Aligning parallel annotations from multiple semantic enumerations for the same sense of a word can significantly increase the lexical knowledge acquired by the model, especially for rare and infrequently used senses.

[0035] To take advantage of this rich lexical and contextual information, annotation alignment, i.e., alignment of the catalogues, can include a best matching function (220) that involves mapping (214, 216) annotations of common words from one of the semantic catalogues to annotations in another of the semantic catalogues such that the word-for-word similarity of the mappings involved in the matching function is maximized.

[0036] In some embodiments, the best matching function (220) can be determined using an optimization setup. In some embodiments, the optimization setup can be Maximum Weighted Bipartite Matching, which aims to find the best matching in a weighted bipartite graph that maximizes the sum of edge weights. As an example, in FIG. 2, the best matching function (220) can treat the annotation mappings (214, 216) as weighted edges, and the best matching function (220) can represent a function that maximizes the weights of the annotation mappings. In some embodiments, the matching function (220) can be configured to maximize the sum of the word-for-word similarities between each annotation of a common word from the first semantic catalog (204-209) and each of the one or more associated annotations from the second semantic catalog (204-209).

[0037] An example of a maximum weighted bipartite matching optimization setup to obtain the best matching function for the alignment of the catalogues may be as follows: We consider two word sets S1 and S2 to be obtained from the semantic catalogue 204 and the semantic catalogue 205, respectively. Each word set consists of a list of definition sentences, i.e., phrase annotations (210, 211, 212). To determine the best matching function (220) f: S1 → S2, we can maximize the reward function r: S1 × S2 → R. In some embodiments, sentence level verbatim similarity or sentence verbatim similarity can be used as the reward function to measure the similarity between two phrase annotations. In some embodiments, a pre-trained secondary model can be used to measure or determine the sentence level verbatim similarity between two phrase annotations. The pre-trained secondary model can be any prior art model capable of performing Semantic Textual Similarity (STS) and Paraphrase Detection tasks. In some embodiments, the Sentence Bidirectional Encoder Representations from Transformers (SBERT) model may be used.

[0038] In some embodiments, determining the sentence verbatim similarities between the phrase annotations may include determining one or more sentence embeddings based on a pre-trained quadratic model. In some embodiments, determining the sentence verbatim similarities between the phrase annotations may include determining a cosine similarity of the phrase annotations based on the sentence verbatim similarities between the phrase annotations. As an example, in some embodiments, a pre-trained quadratic model (e.g., SBERT) may be applied to word sets S1 and S2 to obtain sentence embeddings and compute the cosine similarity as a reward function.

[0039] In some embodiments, the maximum weighted bipartite matching optimization can be solved using linear programming. As an example, a solution to the maximum weighted bipartite matching optimization based on linear programming may be as follows:

[0040] Weight w ij Let x represent the word-for-word similarity between the i-th phrase annotation of S1 and the j-th phrase annotation of S2. The alignment of the semantic catalogs 204 and 205 can be said to involve solving the following linear integer programming problem.

number

[0041] In some embodiments, S1 and S2 may include any of the semantic indices (204-209). In some embodiments, S1 and S2 may include a combination of two of the entire semantic indices (204-209), and the alignment of the indices may include an alignment of all combinations of the semantic indices (204-209). Thus, the alignment of the indices may provide a mapping of the phrase annotation across all of the semantic indices (204-209).

[0042] 3 is an example of a semantic equivalence recognizer model (300) for predicting word meanings according to an embodiment of the present disclosure. The semantic equivalence recognizer model (300) can predict whether a word in a context sentence and a phrase annotation are semantically equivalent. Thus, the semantic equivalence recognizer model (300) predicts the semantic meaning of the context sentence.

[0043] The phrase annotated aligned inventory (also referred to as aligned inventory) (310) can include the mappings of phrase annotations (214, 216) and the best matching function (220). The example phrase annotations (320) can include the mappings of phrase annotations (214, 216) from the aligned inventory. The semantic equivalence recognizer model (300) can be used to predict the meaning of words, and the context sentences can include sentences containing such words.

[0044] According to an embodiment, the semantic equivalence recognizer model (300) can receive example phrase annotations (320) from the annotated aligned inventory (310) as input for training the semantic equivalence recognizer model (300), i.e., for training the transformers (330, 335). In some embodiments, the example phrase annotations (320) from the annotated aligned inventory (310) can be pairs of positive phrase annotations, where the pairs of annotations are aligned. In some embodiments, the example phrase annotations (320) from the annotated aligned inventory (310) can be pairs of negative phrase annotations, where the pairs of annotations are not aligned.

[0045] According to some embodiments, the semantic equivalence recognizer model (300) can include one or more Transformers (330, 335) that predict semantic equivalence between a word in a context and any associated annotations. The Transformers (330, 335) can be deep learning models that include an encoder and a decoder that can handle processing of input data, such as annotations in the example annotations (320) from a sequence using the context used at any position in the input sequence. In some embodiments, the Transformers (330, 335) can be limited to include only an encoder. In some embodiments, the Transformers (330, 335) can be trained using only the example annotations (320).

[0046] In some embodiments, the Transformers (330, 335) (or, more broadly, the semantic equivalence recognizer model (300)) can be trained using augmented training data, which can combine the lexical annotation mappings (214, 216) with built-in training data from a particular semantic lexicon, such as the WSD dataset (315). Thus, with augmented training data, the semantic equivalence recognizer model (300) can be trained simultaneously using both the aligned lexicon and the built-in training data from a particular semantic lexicon, such as the WSD dataset (315).

[0047] In some embodiments, the transformers (330, 335) are first trained using the mappings (214, 216) of the phrase annotations so that the semantic equivalence recognizer model (300) can be a general model that can determine whether words in a context sentence and a phrase annotation are semantically equivalent. However, such models are general and may not properly predict the meaning of domain-specific words. Therefore, the transformers (330, 335) (broadly speaking, the semantic equivalence recognizer model (300)) can be further trained or fine-tuned in the model by connecting the output of the first trained model to additional layers associated with a specific semantic lexicon, such as the WSD dataset (315). This produces a semantic equivalence recognizer model (300) that is specialized in the domain of a specific semantic lexicon, such as the WSD dataset (315). In some embodiments, the specific semantic lexicon used to fine-tune the trained model may be in a different domain than the semantic lexicon used for the aligned lexicon.

[0048] Once trained, the transformer outputs (340, 345) may be determined by the transformers (330, 335), which may include dense representations of the inputs, such as the input example annotations (320) and a semantic representation of the context sentence (325). When the semantic equivalence recognizer model (300) is applied to the context sentence (325), one or more output probabilities (360) are generated by the semantic equivalence recognizer model (300) for one or more annotations that have meanings that are semantically equivalent to the words of the context sentence. The annotation with the highest probability may be selected by the semantic equivalence recognizer model (300) as the predicted meaning for the words of the context sentence.

[0049] FIG. 4 is an exemplary flow diagram of a method 400 for use in a semantic prediction model, according to an embodiment of the present disclosure.

[0050] At 410, one or more semantic lists are used to generate an aligned list. The annotated aligned list (310) can include a best mapping alignment or best matching alignment of annotations across one or more semantic lists. In some embodiments, the semantic lists can be dictionaries that provide multiple example sentences for each sense of a word with its usage, and can be used as a means to receive context sentences for a particular sense. Each semantic list can have multiple examples or annotations of a particular word in a limited number of contexts. Thus, annotations of a word's sense from different semantic lists may be different expressions of the same meaning. Aligning parallel annotations from multiple semantic lists for the same sense can significantly increase the lexical knowledge acquired by the model, especially for rare or infrequently used senses.

[0051] At 420, words of the context sentence can be obtained. The words of the context sentence can be words that have a meaning or semantics that can be predicted by the model. In some embodiments, the entire context sentence can be obtained.

[0052] At 430, one or more semantic equivalence scores can be determined, the one or more semantic equivalence scores indicating a semantic similarity between the words of the context sentence and each of the one or more associated phrase annotations of the one or more aligned lists using the semantic equivalence recognizer model. By way of example, an output probability score may be generated by the semantic equivalence recognizer model (300) indicating a semantic similarity between the words of the context sentence and each of the one or more associated phrase annotations of the one or more aligned lists.

[0053] At 440, predicting the correct meaning of the word of the context sentence can be based on the determined one or more semantic equivalence scores. In some embodiments, predicting the correct meaning of the word of the context sentence based on the determined one or more semantic equivalence scores can include selecting the result phrase annotation associated with the largest semantic equivalence score. As an example, the phrase annotation with the largest probability from the output probabilities generated by the semantic equivalence recognizer model (300) may be selected as the predicted correct meaning of the word of the context sentence.

[0054] FIG. 5 is an exemplary flowchart of a method 500 for using a semantic prediction model to generate one or more aligned catalogs, according to an embodiment of the present disclosure.

[0055] At 510, a phrase annotation can be collected from a first semantic catalogue. By way of example, at 510, a phrase annotation can be collected from a semantic catalogue (204-209) such as a dictionary.

[0056] At 520, the word annotations can be collected from a second semantic catalogue. As an example, at 520, the word annotations can be collected from a dictionary-like semantic catalogue (204-209). In some embodiments, the first and second semantic catalogues can be different.

[0057] At 530, a best match between the first and second semantic libraries can be determined. By way of example, a best matching function (220) may be generated to indicate a mapping (214, 216) of annotations of common words from one of the semantic libraries to annotations of another of the semantic libraries. In some embodiments, the matching function mapping can be generated as a function that maximizes the word-for-word similarity.

[0058] At 540, a word-for-word similarity score between each phrase annotation from the first semantic catalogue and each of the one or more associated phrase annotations from the second semantic catalogue may be determined for each common word between the first and second semantic catalogues. In some embodiments, determining the word-for-word similarity score between each phrase annotation from the first semantic catalogue and each of the one or more associated phrase annotations from the second semantic catalogue may include determining one or more sentence embeddings based on a pre-trained secondary model. In some embodiments, determining the word-for-word similarity score between each phrase annotation from the first semantic catalogue and each of the one or more associated phrase annotations from the second semantic catalogue may include determining a cosine similarity between each phrase annotation from the first semantic catalogue and each of the one or more associated phrase annotations from the second semantic catalogue based on the one or more sentence embeddings.

[0059] At 550, a matching function can be determined. The mapping function can map each phrase annotation from the first semantic catalogue to each of one or more associated phrase annotations from the second semantic catalogue, and the matching function can be configured to maximize a sum of the word-for-word similarity scores between each phrase annotation from the first semantic catalogue and each of the one or more associated phrase annotations from the second semantic catalogue. As an example, the best matching function (220) can be configured to maximize a sum of the word-for-word similarity scores between each phrase annotation from the first semantic catalogue (204) and each of the one or more associated phrase annotations from the second semantic catalogue (205). As another example, the best matching function (220) can be configured to generate a mapping that maximizes the sum of the word-for-word similarity scores.

[0060] At 560, positive phrase annotation pairs may be generated. In some embodiments, based on a determination that a word-for-word similarity score between a phrase annotation from the first semantic catalog and each of the one or more associated phrase annotations from the second semantic catalog exceeds a threshold, a positive phrase annotation pair may be generated by pairing a phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog.

[0061] At 570, negative phrase annotation pairs may be generated. In some embodiments, based on a determination that a word-for-word similarity score between the phrase annotation from the first semantic catalogue and each of the one or more associated phrase annotations from the second semantic catalogue is less than a threshold, the negative phrase annotation pairs are generated by pairing the phrase annotation from the first semantic catalogue with each of the one or more associated phrase annotations from the second semantic catalogue.

[0062] 6 is an exemplary flow chart of a method 600 for use in a semantic prediction model, according to an embodiment of the present disclosure. Process 600 illustrates an exemplary process for determining semantic equivalence scores that indicate semantic similarity.

[0063] At 610, the context sentences can be input to the semantic equivalence recognizer model. At 620, pairs of phrase annotations from the aligned catalog can be input to the semantic equivalence recognizer model. As an example, all positive and negative pairs of phrase annotations from the phrase annotation aligned catalog (310) and the context sentences containing the words whose meanings are to be predicted can be input to the semantic equivalence recognizer model (300).

[0064] At 630, one or more phrase annotations may be identified from one or more aligned lists associated with the words of the context sentence. In some embodiments, phrase annotations associated with words (the context sentence) whose meaning or semantics is to be predicted are identified.

[0065] At 640, a trained annotation classifier may be applied to the identified one or more annotations to generate 650 a probability score for each of the identified one or more annotations.

[0066] In some embodiments, at 645, the phrase annotation classifier can be trained using extended training data, which may be a combination of one or more aligned inventories and built-in training data associated with a particular semantic inventory. As an example, the semantic equivalence recognizer model (300) may be trained using the extended training data. In the case of extended training data, the phrase annotation mappings (214, 216) may be combined with built-in training data of a particular semantic inventory, such as the WSD dataset (315). Thus, using the extended training data, the semantic equivalence recognizer model (300) can be trained using both aligned inventories and built-in training data of a particular semantic inventory, such as the WSD dataset (315), simultaneously.

[0067] 7 is an exemplary flow chart of a method 700 for use in a semantic prediction model, according to an embodiment of the present disclosure. Process 700 illustrates an exemplary process for determining semantic equivalence scores that indicate semantic similarity.

[0068] At 710, the context sentences can be input to the semantic equivalence recognizer model. At 720, pairs of phrase annotations from the aligned catalog can be input to the semantic equivalence recognizer model. As an example, all positive and negative pairs of phrase annotations from the aligned catalog (310) and the context sentences containing the words whose meanings are to be predicted can be input to the semantic equivalence recognizer model (300).

[0069] At 730, one or more phrase annotations may be identified from one or more aligned lists associated with the words of the context sentence. In some embodiments, phrase annotations associated with words (the context sentence) whose meaning or semantics is to be predicted are identified.

[0070] At 740, a trained annotation classifier may be applied to the identified one or more annotations to generate 750 a probability score for each of the identified one or more annotations.

[0071] In some embodiments, the trained phrase annotation classifier may be trained with one or more aligned inventories at 744. In some embodiments, the trained phrase annotation classifier may be fine-tuned with embedded training data associated with a particular semantic inventory of the new domain at 746. As an example, the semantic equivalence recognizer model (300) may be trained first with the phrase annotation mappings (214, 216) so that the semantic equivalence recognizer model (300) may be a general model that can determine whether words in a context sentence and a phrase annotation are semantically equivalent. In some embodiments, the semantic equivalence recognizer model (300) may be further trained or further fine-tuned by connecting the output of the initial trained model to additional layers associated with a particular semantic inventory, such as the WSD dataset (315). This may produce a semantic equivalence recognizer model (300) that is a domain expert in a particular semantic inventory, such as the WSD dataset (315). In some embodiments, the particular semantic lexicon used for fine-tuning in the trained model may be of a different domain than the semantic lexicon used for the aligned lexicon.

[0072] 4-7 show example blocks of processes 400, 500, 600, and 700, in embodiments, processes 400, 500, 600, and 700 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in Figures 4-7. In embodiments, any of the blocks of processes 400, 500, 600, and 700 may be combined or arranged in any total number or order as desired. In embodiments, two or more of the blocks of processes 400, 500, 600, and 700 may be performed in parallel.

[0073] The techniques described above may be implemented as computer software using computer readable instructions physically stored on one or more computer readable media, or by one or more specially configured hardware processors. For example, Figure 10 illustrates a computer system 1000 suitable for implementing various embodiments.

[0074] Computer software may be coded using any suitable machine code or computer language capable of producing code containing instructions that can be executed directly, interpreted, through microcode execution, etc., by a computer's central processing unit (CPU), graphics processing unit (GPU), etc., in accordance with assembly, compilation, linking, or similar mechanisms.

[0075] The instructions may be executed in various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0076] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents, which are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that embody the principles of this disclosure and are therefore within its spirit and scope, even if not explicitly shown or described in this application. [Explanation of symbols]

[0077] 100 Semantic Prediction Models 200 Simplified schematic diagram 204 Semantic Catalog 205 Semantic Catalog 206 Semantic Catalog 207 Semantic Catalog 208 Semantic Catalog 209 Semantic Catalog 210 Phrase Commentary 211 Phrase Commentary 212 Phrase Commentary 214 Mapping 216 Mapping 220 Matching Functions 300 Semantic Equivalence Recognizer Model 310 Annotated and Aligned Catalogue 315 Word Sense Disambiguation (WSD) Dataset 320 Examples of phrase annotations 325 Context sentence 330 Transformers 335 Transformers 340 Transformer Output 345 Transformer Output 360 Output Probability 1000 Computer Systems S1 Word Set S2 Word Set

Claims

1. 1. A method for predicting the meaning of a word, the method being executed by a computer, comprising the steps of: generating one or more aligned inventories using the two or more semantic inventories, collecting word annotations from a first semantic catalog; collecting word annotations from a second semantic catalog; determining a best match between the first semantic bibliography and the second semantic bibliography, the determining the best match between the first semantic bibliography and the second semantic bibliography comprising: determining, for each common word between the first and second semantic libraries, a word-for-word similarity score between each phrase annotation from the first semantic library and each of one or more associated phrase annotations from the second semantic library; determining a matching function that maps each of the phrase annotations from the first semantic catalog to each of the one or more associated phrase annotations from the second semantic catalog, the matching function being configured to maximize a sum of the word-for-word similarity scores between each of the phrase annotations from the first semantic catalog and each of the one or more associated phrase annotations from the second semantic catalog; and generating positive annotation pairs by pairing a phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog based on a determination that the word-for-word similarity score between the phrase annotation from the first semantic catalog and each of the one or more associated phrase annotations from the second semantic catalog exceeds a threshold; generating negative annotation pairs by pairing a phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog based on a determination that the word-for-word similarity score between the phrase annotation from the first semantic catalog and each of the one or more associated phrase annotations from the second semantic catalog is less than the threshold; generating one or more aligned inventories using one or more semantic inventories, obtaining words of a context sentence; determining, using a semantic equivalence recognizer model, one or more semantic equivalence scores indicative of semantic similarity between the words of the context sentence and each of one or more associated phrase annotations of the one or more aligned inventories; predicting a correct sense of the word of the context sentence based on the determined one or more semantic equivalence scores; The method includes:

2. Determining the word-for-word similarity score between each of the word annotations from the first semantic catalog and each of the one or more associated word annotations from the second semantic catalog includes: determining one or more sentence embeddings based on a pre-trained quadratic model; and determining a cosine similarity between each of the phrase annotations from the first semantic bibliography and each of the one or more associated phrase annotations from the second semantic bibliography based on the one or more sentence embeddings; 2. The method of claim 1, comprising:

3. 3. The method of claim 2, wherein the pre-trained secondary model comprises a Sentence Bidirectional Encoder Representations from Transformers (SBERT) model.

4. The step of determining, using the semantic equivalence recognizer model, the one or more semantic equivalence scores indicative of the semantic similarity between the words of the context sentence and each of the one or more associated phrase annotations of the one or more aligned inventories, further comprising: inputting the words of the context sentence into the semantic equivalence recognizer model; inputting the one or more aligned lists into the semantic equivalence recognizer model; identifying one or more lexical annotations from the one or more aligned lists associated with the words of the context sentence; applying a trained annotation classifier to the one or more identified annotations to generate a probability score for each of the one or more identified annotations; 2. The method of claim 1, comprising:

5. 5. The method of claim 4, wherein the trained phrase annotation classifier is trained using extended training data, the extended training data being a combination of the one or more aligned inventories and embedded training data associated with a particular semantic inventory.

6. 5. The method of claim 4, wherein the trained phrase annotation classifier is trained using the one or more aligned inventories, and the trained phrase annotation classifier is fine-tuned using embedded training data associated with a specific semantic inventory of a new domain.

7. The method of claim 1 , wherein the one or more semantic inventories are lexical datasets for a language.

8. 2. The method of claim 1 , wherein predicting the correct meaning of the words of the context sentence based on the determined one or more semantic equivalence scores comprises selecting a result phrase annotation associated with a maximum semantic equivalence score.

9. An apparatus for predicting word meaning, the apparatus comprising: at least one memory configured to store program code; at least one processor configured to read the program code and perform operations as instructed by the program code; wherein the program code comprises: a first generation code configured to cause the at least one processor to generate one or more aligned inventories using one or more semantic inventories, the first generation code comprising: a first collection code configured to cause the at least one processor to collect lexical annotations from a first semantic bibliography; a second collection code configured to cause the at least one processor to collect lexical annotations from a second semantic bibliography; a second decision code configured to cause the at least one processor to determine a best match between the first semantic lexicon and the second semantic lexicon, the second decision code comprising: third decision code configured to cause the at least one processor to determine, for each common word in the first semantic catalogue and the second semantic catalogue, a word-for-word similarity score between each phrase annotation from the first semantic catalogue and each of one or more associated phrase annotations from the second semantic catalogue; a fourth decision code configured to cause the at least one processor to determine a matching function that maps each of the phrase annotations from the first semantic catalog to each of the one or more associated phrase annotations from the second semantic catalog, the matching function configured to maximize a sum of the word-for-word similarity scores between each of the phrase annotations from the first semantic catalog and each of the one or more associated phrase annotations from the second semantic catalog; a second decision code further comprising: second generation code configured to cause the at least one processor to generate positive annotation pairs by pairing a phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog based on a determination that the word-for-word similarity score between the phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog exceeds a threshold; and third generation code configured to cause the at least one processor to generate negative annotation pairs by pairing a phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog based on a determination that the word-for-word similarity score between the phrase annotation from the first semantic catalog with each of the one or more associated phrase annotations from the second semantic catalog is less than the threshold. a first obtaining code configured to cause the at least one processor to obtain words of a context sentence; a first decision code configured to cause the at least one processor to determine, using a semantic equivalence recognizer model, one or more semantic equivalence scores indicative of a semantic similarity between the words of the context sentence and each of one or more associated phrase annotations of the one or more aligned inventories; a first prediction code configured to cause the at least one processor to predict a correct meaning of the word of the context sentence based on the determined one or more semantic equivalence scores; and 13. An apparatus comprising:

10. The third decision code is fifth decision code configured to cause the at least one processor to determine one or more sentence embeddings based on a pre-trained secondary model; and a sixth decision code configured to cause the at least one processor to determine a cosine similarity between each of the phrase annotations from the first semantic bibliography and each of the one or more associated phrase annotations from the second semantic bibliography based on the one or more sentence embeddings; The apparatus of claim 9, further comprising:

11. The first decision code is a first input code configured to cause the at least one processor to input the words of the context sentence to the semantic equivalence recognizer model; second input code configured to cause the at least one processor to input the one or more aligned inventories to the semantic equivalence recognizer model; a first identification code configured to cause the at least one processor to identify one or more lexical annotations from the one or more aligned lists associated with the words of the context sentence; a first applying code configured to cause the at least one processor to apply a trained annotation classifier to the identified one or more annotations to generate a probability score for each of the identified one or more annotations; The apparatus of claim 9, further comprising:

12. 12. The apparatus of claim 11, wherein the trained phrase annotation classifier is trained using extended training data, the extended training data being a combination of the one or more aligned inventories and embedded training data associated with a particular semantic inventory.

13. 13. The apparatus of claim 12, wherein the trained phrase annotation classifier is trained using the one or more aligned inventories, and the trained phrase annotation classifier is fine-tuned using embedded training data associated with a specific semantic inventory of a new domain.

14. The apparatus of claim 9 , wherein the one or more semantic libraries are a lexical dataset for a language.

15. 10. The apparatus of claim 9, wherein the first prediction code further comprises a first selection code configured to cause the at least one processor to select a result phrase annotation associated with a maximum semantic equivalence score.

16. A program for causing one or more processors to carry out the method according to any one of claims 1 to 8.

Images