Chinese vocabulary disambiguation method, device, terminal equipment and storage medium

Abstract

The present invention application provides a Chinese vocabulary disambiguation method, apparatus, terminal device and storage medium, the method comprising: obtaining a target Chinese vocabulary, obtaining a long interpretation word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, wherein the long interpretation word vector expression is determined according to a preset mapping relationship; respectively calculating the similarity between the polysemous word vector expression and each long interpretation word vector expression, and disambiguating the target Chinese vocabulary in the target sentence according to the long interpretation word vector expression with the highest similarity. Compared with the technical solution of the prior art that solely relies on the Chinese knowledge base, the present invention application can enhance the Chinese knowledge base through a foreign language knowledge base, improve its richness and refinement, make up for the lack of resources in the Chinese knowledge base, and thus improve the accuracy of the interpretation.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a Chinese vocabulary disambiguation method, device, terminal equipment and storage medium. Background Art

[0002] With the large-scale application of artificial intelligence in various fields, when the intelligent agent determines the target call word, it needs to have a certain degree of understanding of the word, or needs to show the meaning of the target word to the user. In the Chinese vocabulary, factors such as polysemy can lead to ambiguity in the interpretation of the target word. Unsupervised Chinese word sense disambiguation uses an external vocabulary knowledge base, a trained language model, and contextual content to determine the exact meaning of a specified word in a Chinese sentence. This method does not rely on large-scale annotated data.

[0003] At present, the unsupervised disambiguation methods on the market mainly rely on the knowledge base of Chinese itself. However, the resources of the current Chinese knowledge base are still insufficient. The Chinese knowledge base lacks sufficient details and extensive corpus support. This deficiency leads to the limitation of unsupervised Chinese word meaning representation and the low accuracy of the meaning interpretation of polysemous words. Summary of the invention

[0004] The present invention provides a Chinese vocabulary disambiguation method, device, terminal equipment and storage medium, which reduce the dependence on a Chinese knowledge base and improve the accuracy of interpretation.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a Chinese vocabulary disambiguation method, comprising:

[0006] Acquire a target Chinese vocabulary, and obtain a long interpretation word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, wherein the long interpretation word vector expression is determined according to a preset mapping relationship, and the preset mapping relationship maps at least one foreign language to the same word vector space;

[0007] The similarity between the polysemous word vector expression and each long interpretation word vector expression is calculated respectively, and the target interpretation of the target Chinese vocabulary in the target sentence is obtained according to the long interpretation word vector expression with the highest similarity; and the target Chinese vocabulary in the target sentence is disambiguated according to the target interpretation.

[0008] As a preferred solution, the step of obtaining the long meaning word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence based on the target Chinese vocabulary and the pre-trained cross-language model includes:

[0009] Acquire a plurality of foreign language corpora corresponding to the target Chinese vocabulary; the foreign language corpora contain a plurality of definitions;

[0010] Inputting the foreign language corpus into the cross-language model;

[0011] Based on the output of the cross-language model, a first label and a plurality of second labels corresponding to the target Chinese vocabulary are obtained, wherein the first label is a label corresponding to the target interpretation, and the second label is a hidden layer label of the position of the target Chinese vocabulary in the target sentence;

[0012] Filter out vectors of the foreign language corpus corresponding to the first label and vectors of the foreign language corpus corresponding to the second label; perform a first average processing on the vectors of the foreign language corpus corresponding to the filtered first label, and obtain the vector expression of the long interpretation word based on the first average processing result; perform a second average processing based on the vectors of the foreign language corpus corresponding to each of the second labels, and obtain the polysemous word vector expression of the target Chinese vocabulary in the target sentence based on the second average processing result.

[0013] As a preferred solution, before obtaining the first label and multiple second labels corresponding to the target Chinese vocabulary, the method further includes:

[0014] Performing word segmentation processing on the foreign language corpus to obtain a word list;

[0015] Initialize the corresponding word vector according to the word list;

[0016] Analyzing context information of each word in the word list using word co-occurrence information in a preset corpus, wherein the context information includes co-occurrence relationships of words in the context;

[0017] Based on the co-occurrence relationships, a co-occurrence matrix between words is obtained; and the co-occurrence matrix is ​​decomposed to obtain an initial word vector representation;

[0018] The initial word vector representation is subjected to multi-level clustering; the part of speech and dependency relationship of each word in the word list are fused with the clustering result to obtain an enhanced word vector representation of the foreign language corpus.

[0019] As a preferred solution, the preset mapping relationship includes a first mapping relationship or a second mapping relationship; before obtaining the long meaning word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence, it also includes:

[0020] Acquire multiple synonym sets and multiple concept word sets corresponding to the target Chinese vocabulary, wherein the synonym sets include foreign language synonyms and Chinese synonyms, and the concept word sets include foreign language words and Chinese words, the synonym sets are acquired from a foreign language knowledge base, and the concept word sets are acquired from a Chinese knowledge base;

[0021] When a foreign language word is consistent with a foreign language synonym, or when a Chinese word is consistent with a Chinese synonym, a first mapping relationship is constructed between the corresponding synonym set and the corresponding concept word set; otherwise, the overlap between each synonym set and each concept word set is obtained, and a second mapping relationship is constructed between the synonym set with the highest overlap and the concept word set.

[0022] As a preferred solution, the calculation formula of the overlap degree is:

[0023] ;

[0024] in, is the overlap between A and B, A is the synonym set, and B is the concept word set.

[0025] As a preferred solution, the obtaining of the overlap between each synonym set and each concept word set includes:

[0026] Get the number of words in the synonym set and the number of words in the concept set;

[0027] According to the number of words in the synonym set and the number of words in the concept word set, the number of overlapping words is determined;

[0028] The overlap ratio is calculated according to the number of overlapped words; the overlap ratio includes a first ratio of overlapped words in the synonym set and a second ratio of overlapped words in the concept word set;

[0029] When both the first proportion and the second proportion are greater than a preset ratio value, determining the proportion of each part of speech in the synonym set and the proportion of each part of speech in the concept word set according to the corpus corresponding to the synonym set and the corpus corresponding to the concept word set;

[0030] The synonym set is sorted according to the proportion from large to small to obtain a first sorting result; the concept word set is sorted according to the proportion from large to small to obtain a second sorting result;

[0031] Verifying the first sorting result and the second sorting result; when the two are consistent, verifying the overlapping words through a preset concept tree structure;

[0032] When the verification is passed, the degree of overlap between the synonym set and the concept word set is determined according to the overlap ratio.

[0033] As a preferred solution, the similarity between the polysemous word vector expression and each long-meaning word vector expression is calculated separately, including:

[0034] The similarity is calculated according to the following formula:

[0035] ;

[0036] Among them, Sim(D i ,D j ) is the long meaning word vector D i And the polysemous word vector D j The similarity between them, n refers to the dimension of the quantity, t ik is the long meaning word vector D i The kth number in t jk D is the polysemous word vector j The kth number in .

[0037] Accordingly, the present invention provides a Chinese vocabulary disambiguation device, comprising a first determination module and an interpretation determination module; wherein:

[0038] The first determination module is used to obtain a target Chinese vocabulary, and obtain a long interpretation word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, wherein the long interpretation word vector expression is determined according to a preset mapping relationship, and the preset mapping relationship maps at least one foreign language to the same word vector space;

[0039] The interpretation determination module is used to respectively calculate the similarity between the polysemous word vector expression and each long interpretation word vector expression, and obtain the target interpretation of the target Chinese vocabulary in the target sentence based on the long interpretation word vector expression with the highest similarity; and disambiguate the target Chinese vocabulary in the target sentence based on the target interpretation.

[0040] As a preferred solution, the first determination module obtains the long meaning word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence according to the target Chinese vocabulary and the pre-trained cross-language model, including:

[0041] The first determination module obtains a number of foreign language corpora corresponding to the target Chinese vocabulary; the foreign language corpora contain a plurality of definitions;

[0042] Inputting the foreign language corpus into the cross-language model;

[0043] Based on the output of the cross-language model, a first label and a plurality of second labels corresponding to the target Chinese vocabulary are obtained, wherein the first label is a label corresponding to the target interpretation, and the second label is a hidden layer label of the position of the target Chinese vocabulary in the target sentence;

[0044] Filter out vectors of the foreign language corpus corresponding to the first label and vectors of the foreign language corpus corresponding to the second label; perform a first average processing on the vectors of the foreign language corpus corresponding to the filtered first label, and obtain the vector expression of the long interpretation word based on the first average processing result; perform a second average processing based on the vectors of the foreign language corpus corresponding to each of the second labels, and obtain the polysemous word vector expression of the target Chinese vocabulary in the target sentence based on the second average processing result.

[0045] As a preferred solution, the Chinese vocabulary disambiguation device further includes a vector acquisition module; the vector acquisition module is used for: before the first determination module obtains the first label and multiple second labels corresponding to the target Chinese vocabulary:

[0046] Performing word segmentation processing on the foreign language corpus to obtain a word list;

[0047] Initialize the corresponding word vector according to the word list;

[0048] Analyzing context information of each word in the word list using word co-occurrence information in a preset corpus, wherein the context information includes co-occurrence relationships of words in the context;

[0049] Based on the co-occurrence relationships, a co-occurrence matrix between words is obtained; and the co-occurrence matrix is ​​decomposed to obtain an initial word vector representation;

[0050] The initial word vector representation is subjected to multi-level clustering; the part of speech and dependency relationship of each word in the word list are fused with the clustering result to obtain an enhanced word vector representation of the foreign language corpus.

[0051] As a preferred embodiment, the preset mapping relationship includes a first mapping relationship or a second mapping relationship; the Chinese vocabulary disambiguation device also includes a mapping configuration module; the mapping configuration module is used for obtaining the long interpretation word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence before the first determination module obtains:

[0052] Acquire multiple synonym sets and multiple concept word sets corresponding to the target Chinese vocabulary, wherein the synonym sets include foreign language synonyms and Chinese synonyms, and the concept word sets include foreign language words and Chinese words, the synonym sets are acquired from a foreign language knowledge base, and the concept word sets are acquired from a Chinese knowledge base;

[0053] When a foreign language word is consistent with a foreign language synonym, or when a Chinese word is consistent with a Chinese synonym, a first mapping relationship is constructed between the corresponding synonym set and the corresponding concept word set; otherwise, the overlap between each synonym set and each concept word set is obtained, and a second mapping relationship is constructed between the synonym set with the highest overlap and the concept word set.

[0054] As a preferred solution, the calculation formula of the overlap degree is:

[0055] ;

[0056] in, is the overlap between A and B, A is the synonym set, and B is the concept word set.

[0057] As a preferred solution, the mapping configuration module obtains the overlap between each synonym set and each concept word set, including:

[0058] The mapping configuration module obtains the number of words in the synonym set and the number of words in the concept word set;

[0059] According to the number of words in the synonym set and the number of words in the concept word set, the number of overlapping words is determined;

[0060] The overlap ratio is calculated according to the number of overlapped words; the overlap ratio includes a first ratio of overlapped words in the synonym set and a second ratio of overlapped words in the concept word set;

[0061] When both the first proportion and the second proportion are greater than a preset ratio value, determining the proportion of each part of speech in the synonym set and the proportion of each part of speech in the concept word set according to the corpus corresponding to the synonym set and the corpus corresponding to the concept word set;

[0062] The synonym set is sorted according to the proportion from large to small to obtain a first sorting result; the concept word set is sorted according to the proportion from large to small to obtain a second sorting result;

[0063] Verifying the first sorting result and the second sorting result; when the two are consistent, verifying the overlapping words through a preset concept tree structure;

[0064] When the verification is passed, the degree of overlap between the synonym set and the concept word set is determined according to the overlap ratio.

[0065] As a preferred solution, the interpretation determination module calculates the similarity between the polysemous word vector expression and each long interpretation word vector expression respectively, including:

[0066] The interpretation determination module calculates the similarity according to the following formula:

[0067] ;

[0068] Among them, Sim(D i ,D j ) is the long meaning word vector D i And the polysemous word vector D jThe similarity between them, n refers to the dimension of the quantity, t ik is the long meaning word vector D i The kth number in t jk D is the polysemous word vector j The kth number in .

[0069] Correspondingly, the present invention application provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the Chinese vocabulary disambiguation method when executing the computer program.

[0070] Accordingly, the present invention application provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the Chinese vocabulary disambiguation method.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] The present invention application provides a Chinese vocabulary disambiguation method, apparatus, terminal device and storage medium, the Chinese vocabulary disambiguation method comprising: obtaining a target Chinese vocabulary, obtaining a long interpretation word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, the long interpretation word vector expression being determined according to a preset mapping relationship, the preset mapping relationship mapping at least one foreign language to the same word vector space; respectively calculating the similarity between the polysemous word vector expression and each long interpretation word vector expression, obtaining a target interpretation of the target Chinese vocabulary in the target sentence according to the long interpretation word vector expression with the highest similarity; and disambiguating the target Chinese vocabulary in the target sentence according to the target interpretation. The present invention application determines the long interpretation word vector expression according to a preset mapping relationship, and the mapping relationship maps at least one foreign language to the same word vector space. Compared with the technical solutions of the prior art that simply rely on the Chinese knowledge base, the Chinese knowledge base can be enhanced through the foreign language knowledge base, thereby improving its richness and refinement, making up for the lack of Chinese knowledge base resources, and thus improving the accuracy of the interpretation; in addition, the present invention application obtains the long interpretation word vector expression and the polysemous word vector expression of the target Chinese vocabulary in the target sentence through a cross-language model, thereby obtaining the long interpretation word vector expression with the highest similarity. Compared with the method of obtaining short interpretations in the prior art, the accuracy of the polysemous word interpretation can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 : A flow chart of an embodiment of the Chinese vocabulary disambiguation method provided in the present application.

[0074] Figure 2 : A flow chart of another embodiment of the Chinese vocabulary disambiguation method provided in the present application.

[0075] Figure 3 : A schematic diagram of a framework of an embodiment of the Chinese vocabulary disambiguation method provided in the present application.

[0076] Figure 4 : A schematic diagram of the principles of an embodiment of the Chinese vocabulary disambiguation method provided in the present application.

[0077] Figure 5 : A structural schematic diagram of an embodiment of a Chinese vocabulary disambiguation device provided in the present application. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] Embodiment one:

[0080] Please refer to Figure 1 , Figure 1 A Chinese vocabulary disambiguation method provided by an embodiment of the present invention includes steps S101 to S102; wherein:

[0081] Step S101, obtaining a target Chinese vocabulary, and obtaining a long-meaning word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence based on the target Chinese vocabulary and a pre-trained cross-language model.

[0082] In this step, the long interpretation word vector expression is determined according to a preset mapping relationship, and the preset mapping relationship maps at least one foreign language to the same word vector space. The number of foreign languages ​​selected by the preset mapping relationship is determined according to the parameters of the cross-language model, and the parameters of the cross-language model are determined according to the training situation.

[0083] In the related technology, traditional methods mainly use specific language models to represent word vectors. However, these methods are limited by the resources of the Chinese knowledge base and cannot make full use of external knowledge bases in other languages ​​to obtain more accurate and detailed interpretations of polysemous words. They usually obtain brief interpretations of words, such as synonyms, antonyms, senses, and semes.

[0084] In this step, the long meaning word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence are obtained through a pre-trained cross-language model, and the long meaning word vector expression is determined according to a preset mapping relationship, which can map at least one foreign language to the same word vector space, so that sentences in different languages ​​can be used in the space to perform the similarity calculation shown in step S102, so that the rich resources of the foreign language knowledge base can be used to enhance the Chinese knowledge base, and the richness and refinement of the Chinese knowledge base can be improved, which effectively solves the problem of insufficient resources in the existing Chinese knowledge base and ensures the accuracy of obtaining the target Chinese vocabulary interpretation in subsequent steps.

[0085] Exemplarily, the present invention application may select the XLM-RoBERTa cross-language model. This model is based on the RoBERTa (Robustly optimized BERT approach) structure, which is essentially a variant of BERT for cross-language natural language processing tasks. In addition, the XLM-RoBERTa cross-language model adopts the Transformer architecture, which includes multiple self-attention layers, each of which consists of a multi-head self-attention mechanism and a feedforward neural network. The XLM-RoBERTa pre-trained model is applicable to more than 100 languages, designed to process multilingual texts and improve cross-language natural language understanding capabilities.

[0086] In some embodiments, Figure 2 As shown, the long-meaning word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence are obtained according to the target Chinese vocabulary and the pre-trained cross-language model, including:

[0087] Step S201, obtaining a plurality of foreign language corpora corresponding to the target Chinese vocabulary; the foreign language corpora contain a plurality of definitions.

[0088] Step S202: input the foreign language corpus into the cross-language model.

[0089] Step S203, based on the output of the cross-language model, obtain a first label and multiple second labels corresponding to the target Chinese vocabulary, wherein the first label is a label corresponding to the target interpretation, and the second label is a hidden layer label of the position of the target Chinese vocabulary in the target sentence.

[0090] Step S204, filter out the vectors of the foreign language corpus corresponding to the first label and the vectors of the foreign language corpus corresponding to the second label; perform a first average processing on the vectors of the foreign language corpus corresponding to the filtered first label, and obtain the vector expression of the long interpretation word based on the first average processing result; perform a second average processing based on the vectors of the foreign language corpus corresponding to each of the second labels, and obtain the polysemous word vector expression of the target Chinese vocabulary in the target sentence based on the second average processing result.

[0091] As an example of this implementation, the model framework used in this implementation can be as follows: Figure 3 shown.

[0092] in, Figure 3 The Cross-language pre-training models shown refer to the pre-trained cross-language models of this application. Long Interpretation Embedding 1 refers to the long interpretation word vector expression mentioned above, and Long Interpretation Embedding 2 refers to the polysemous word vector expression mentioned above. "It's several degrees below zero outside the greenhouse, and the flowers and vegetables are fresh and green inside the greenhouse" is the target Chinese vocabulary described in this application. Target Word Embedding is the target interpretation of the target Chinese vocabulary in the target sentence.

[0093] This example obtains several foreign language corpora corresponding to the target Chinese vocabulary, such as "Aparticular item of prepared food", "Edible seeds or roots or stems or leaves or bulbs or tubers or nonsweet fruits of any of numerous herbaceous plant", etc. These corpora are expressed in the form of English sentences en_glosses. These corpora are input into the cross-language training model, and based on the output of the cross-language model, the first label (first label) of the last hidden layer of each glosses is obtained; the vector of the foreign language corpus corresponding to the obtained first label is subjected to the first average processing to obtain the long interpretation word vector expression. At the same time, the last four hidden layer labels (second labels) of the target polysemous word position are extracted in the target sentence for the second average processing to obtain the word vector expression representing the polysemous word in the target sentence, that is, the polysemous word vector expression.

[0094] When a foreign language corpus is input into a cross-language model, the cross-language model converts the corpus into a representation of a digital vector. For example, as a preferred implementation, before obtaining the first label and multiple second labels corresponding to the target Chinese vocabulary:

[0095] The foreign language corpus is segmented to obtain a word list; the corresponding word vector is initialized according to the word list; the context information of each word in the word list is analyzed by using the word co-occurrence information in a preset corpus, and the context information includes the co-occurrence relationship of the words in the context; based on each of the co-occurrence relationships, a co-occurrence matrix between the words is obtained; the co-occurrence matrix is ​​decomposed to obtain an initial word vector representation; the initial word vector representation is clustered at multiple levels; the part of speech and dependency relationship of each word in the word list are integrated with the clustering result to obtain an enhanced word vector representation of the foreign language corpus.

[0096] In implementing this preferred embodiment, when the input words are segmented, Jieba word segmentation library (Jieba) can be used. First, a Chinese text is input, such as "Natural language processing is an important branch of artificial intelligence". After word segmentation, a preliminary word list ["natural language processing", "is", "artificial intelligence", "of", "important", "branch"] is obtained. Then, the word co-occurrence information in the corpus is used to analyze the co-occurrence relationship of words in different contexts to obtain a co-occurrence matrix between words. For example, assuming that there are 50 words, the number of co-occurrences of each word is obtained by counting the contexts with a window size of 5. The co-occurrence matrix can be expressed as a 50x50 matrix, where (ij) represents the number of co-occurrences of word i and word j. Furthermore, the results of part-of-speech tagging and dependency syntactic analysis are converted into additional vector representations and spliced ​​with word vectors to obtain an enhanced word vector representation. This word vector representation method integrates grammatical attributes and can effectively improve the accuracy and effect of semantic understanding.

[0097] As a preferred implementation, the preset mapping relationship includes a first mapping relationship or a second mapping relationship. Before obtaining the long-meaning word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence, it also includes:

[0098] Acquire multiple synonym sets and multiple concept word sets corresponding to the target Chinese vocabulary, wherein the synonym sets include foreign language synonyms and Chinese synonyms, and the concept word sets include foreign language words and Chinese words, the synonym sets are acquired from a foreign language knowledge base, and the concept word sets are acquired from a Chinese knowledge base; compare each synonym set with each concept word set, and when the foreign language words are consistent with the foreign language synonyms, or when the Chinese words are consistent with the Chinese synonyms, construct a first mapping relationship between the corresponding synonym set and the corresponding concept word set; otherwise, obtain the overlap between each synonym set and each concept word set, and construct a second mapping relationship between the synonym set with the highest overlap and the concept word set.

[0099] For example, this embodiment can use the English knowledge base BabelNet and the Chinese knowledge base HowNet, such as Figure 4 As shown in the figure. A word has multiple Synsets (synonym sets) in the English knowledge base BabelNet, and each Synset (synonym set) consists of multiple Sememes (semantic primitives); each Synset (synonym set) can also be represented by multiple English glosses; similarly, a word has multiple Senses (concept sets described in this application) in the Chinese knowledge base HowNet, and each Sense (concept set) consists of multiple Sememes (semantic primitives).

[0100] From the above, it can be determined whether the zh_word (Chinese word) and en_word (English word) of each Sense (concept) of a word are consistent with the zh_synonyms (Chinese synonyms) and en_synonyms (English synonyms) in the Synset (synonym set).

[0101] When the zh_word (Chinese word) is consistent with the zh_synonyms (Chinese synonyms), or the en_word (English word) is consistent with the en_synonyms (English synonyms), a mapping relationship between the Sense (concept set) and the found Synset (synonym set) is constructed as the first mapping relationship.

[0102] When the condition of "the zh_word (Chinese word) is consistent with the zh_synonyms (Chinese synonyms), or the en_word (English word) is consistent with the en_synonyms (English synonyms)" is not satisfied, the overlap degree of the semantic primitives in the synonym set and the semantic primitives in the concept set can be calculated, and the mapping relationship between the synonym set and the concept set with the highest overlap degree is constructed to obtain the second mapping relationship.

[0103] As an example of this embodiment, the overlap degree can be the intersection of the synonym set and the concept set divided by the union of the synonym set and the concept set. For example, the calculation formula for the overlap degree can be:

[0104] ;

[0105] where is the overlap degree between A and B, A is the synonym set, and B is the concept set.

[0106] For example, in Figure 4 in HowNet, vegetable|蔬菜 has 1 Sememe that overlaps with BabelNet and no overlapping Sememe with dish|菜肴, then the mapping relationship between the two is constructed.

[0107] As another example of this implementation, the obtaining of the overlap between each synonym set and each concept word set includes:

[0108] The number of words in the synonym set and the number of words in the concept word set are obtained; the number of overlapping words is determined according to the number of words in the synonym set and the number of words in the concept word set; the overlap ratio is calculated according to the number of overlapping words; the overlap ratio includes a first proportion of overlapping words in the synonym set and a second proportion of overlapping words in the concept word set; when the first proportion and the second proportion are both greater than a preset proportion value, the proportion of each type of part of speech in the synonym set and the proportion of each type of part of speech (such as noun, verb, adjective or adverb, etc.) in the concept word set are determined respectively according to the corpus corresponding to the synonym set and the corpus corresponding to the concept word set; the various types of parts of speech in the synonym set are sorted from large to small according to the proportion, and a first sorting result is obtained; the various types of parts of speech in the concept word set are sorted from large to small according to the proportion, and a second sorting result is obtained; the first sorting result and the second sorting result are verified; when the two are consistent, the overlapping words are verified through a preset concept tree structure; when the verification is passed, the overlap degree between the synonym set and the concept word set is determined according to the overlap ratio.

[0109] This implementation verifies the first sorting result and the second sorting result, and checks the overlapping words based on a preset concept tree structure. When the verification and check are passed, the overlap degree between the synonym set and the concept word set is obtained, which can ensure the reliability of the overlap degree data finally calculated and ensure that the overlap degree data takes into account dimensions such as part of speech.

[0110] Step S102, respectively calculating the similarity between the polysemous word vector expression and each long interpretation word vector expression, obtaining the target interpretation of the target Chinese vocabulary in the target sentence based on the long interpretation word vector expression with the highest similarity; and disambiguating the target Chinese vocabulary in the target sentence based on the target interpretation.

[0111] In this step, the similarity is calculated as Figure 3 As shown in "Similarity compute", the target interpretation of the target Chinese word in the target sentence can be obtained according to the long interpretation word vector expression with the highest similarity; and the target Chinese word in the target sentence is disambiguated according to the target interpretation. The similarity calculation formula can ensure that the target interpretation of the target Chinese word in the target sentence is the most accurate result in the calculation results, reducing the deviation from the actual meaning.

[0112] In some preferred implementations, the similarity calculation formula includes:

[0113] ;

[0114] Among them, Sim(D i ,D j ) is the long meaning word vector D i And the polysemous word vector D j The similarity between them, n refers to the dimension of the quantity, t ik is the long meaning word vector D i The kth number in t jk D is the polysemous word vector j When using the bert model, n=768; when using the xlm-roberta model, n is 1024.

[0115] Based on one or more of the above-mentioned embodiments, the present application also provides an application example of applying the above-mentioned Chinese vocabulary disambiguation method, as shown in Table 1 and Table 2.

[0116]

[0117] Table 1

[0118]

[0119] Table 2

[0120] On the same data set, the overall micro F1 value (micro-F1 shown in the table) of this application example increased by 4.65%, and the overall macro F1 value (macro-F1 shown in the table) increased by 5.87%, and the performance was improved. In the table, Group refers to the group, Model refers to the interpretation determination model (corresponding to the Chinese vocabulary disambiguation method of this application), Nouns refers to nouns, Verbs refers to verbs, and Overall refers to the overall result of combining verbs and nouns. In addition, the application example shown in the above table also compares other multilingual pre-training models such as bembert, but since this model does not truly project different languages ​​into the same word vector space for expression, the information between languages ​​is not interoperable, and there is no shared knowledge between different language models, so the performance is not as good as the Chinese vocabulary disambiguation method of this application.

[0121] Accordingly, refer to Figure 5 The present invention provides a Chinese vocabulary disambiguation device, including a first determination module 501 and an interpretation determination module 502; wherein,

[0122] The first determination module 501 is used to obtain a target Chinese vocabulary, and obtain a long interpretation word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, wherein the long interpretation word vector expression is determined according to a preset mapping relationship, and the preset mapping relationship maps at least one foreign language to the same word vector space;

[0123] The interpretation determination module 502 is used to respectively calculate the similarity between the polysemous word vector expression and each long interpretation word vector expression, and obtain the target interpretation of the target Chinese vocabulary in the target sentence based on the long interpretation word vector expression with the highest similarity; and disambiguate the target Chinese vocabulary in the target sentence based on the target interpretation.

[0124] As a preferred solution, the first determination module 501 obtains the long meaning word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence according to the target Chinese vocabulary and the pre-trained cross-language model, including:

[0125] The first determination module 501 obtains a number of foreign language corpora corresponding to the target Chinese vocabulary; the foreign language corpora contain a plurality of definitions;

[0126] Inputting the foreign language corpus into the cross-language model;

[0127] Based on the output of the cross-language model, a first label and a plurality of second labels corresponding to the target Chinese vocabulary are obtained, wherein the first label is a label corresponding to the target interpretation, and the second label is a hidden layer label of the position of the target Chinese vocabulary in the target sentence;

[0128] Filter out vectors of the foreign language corpus corresponding to the first label and vectors of the foreign language corpus corresponding to the second label; perform a first average processing on the vectors of the foreign language corpus corresponding to the filtered first label, and obtain the vector expression of the long interpretation word based on the first average processing result; perform a second average processing based on the vectors of the foreign language corpus corresponding to each of the second labels, and obtain the polysemous word vector expression of the target Chinese vocabulary in the target sentence based on the second average processing result.

[0129] As a preferred solution, the Chinese vocabulary disambiguation device further includes a vector acquisition module; the vector acquisition module is used for: before the first determination module obtains the first label and multiple second labels corresponding to the target Chinese vocabulary:

[0130] Performing word segmentation processing on the foreign language corpus to obtain a word list;

[0131] Initialize the corresponding word vector according to the word list;

[0132] Analyzing context information of each word in the word list using word co-occurrence information in a preset corpus, wherein the context information includes co-occurrence relationships of words in the context;

[0133] Based on the co-occurrence relationships, a co-occurrence matrix between words is obtained; and the co-occurrence matrix is ​​decomposed to obtain an initial word vector representation;

[0134] The initial word vector representation is subjected to multi-level clustering; the part of speech and dependency relationship of each word in the word list are fused with the clustering result to obtain an enhanced word vector representation of the foreign language corpus.

[0135] As a preferred embodiment, the preset mapping relationship includes a first mapping relationship or a second mapping relationship; the Chinese vocabulary disambiguation device also includes a mapping configuration module; the mapping configuration module is used for obtaining the long interpretation word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence before the first determination module obtains:

[0136] Acquire multiple synonym sets and multiple concept word sets corresponding to the target Chinese vocabulary, wherein the synonym sets include foreign language synonyms and Chinese synonyms, and the concept word sets include foreign language words and Chinese words, the synonym sets are acquired from a foreign language knowledge base, and the concept word sets are acquired from a Chinese knowledge base;

[0137] When a foreign language word is consistent with a foreign language synonym, or when a Chinese word is consistent with a Chinese synonym, a first mapping relationship is constructed between the corresponding synonym set and the corresponding concept word set; otherwise, the overlap between each synonym set and each concept word set is obtained, and a second mapping relationship is constructed between the synonym set with the highest overlap and the concept word set.

[0138] As a preferred solution, the calculation formula of the overlap degree is:

[0139] ;

[0140] in, is the overlap between A and B, A is the synonym set, and B is the concept word set.

[0141] As a preferred solution, the mapping configuration module obtains the overlap between each synonym set and each concept word set, including:

[0142] The mapping configuration module obtains the number of words in the synonym set and the number of words in the concept word set;

[0143] According to the number of words in the synonym set and the number of words in the concept word set, the number of overlapping words is determined;

[0144] The overlap ratio is calculated according to the number of overlapped words; the overlap ratio includes a first ratio of overlapped words in the synonym set and a second ratio of overlapped words in the concept word set;

[0145] When both the first proportion and the second proportion are greater than a preset ratio value, determining the proportion of each part of speech in the synonym set and the proportion of each part of speech in the concept word set according to the corpus corresponding to the synonym set and the corpus corresponding to the concept word set;

[0146] The synonym set is sorted according to the proportion from large to small to obtain a first sorting result; the concept word set is sorted according to the proportion from large to small to obtain a second sorting result;

[0147] Verifying the first sorting result and the second sorting result; when the two are consistent, verifying the overlapping words through a preset concept tree structure;

[0148] When the verification is passed, the degree of overlap between the synonym set and the concept word set is determined according to the overlap ratio.

[0149] As a preferred solution, the interpretation determination module 502 calculates the similarity between the polysemous word vector expression and each long interpretation word vector expression respectively, including:

[0150] The interpretation determination module 502 calculates the similarity according to the following formula:

[0151] ;

[0152] Among them, Sim(D i ,D j ) is the long meaning word vector D i And the polysemous word vector D j The similarity between them, n refers to the dimension of the quantity, t ik is the long meaning word vector D i The kth number in t jk D is the polysemous word vector j The kth number in .

[0153] Correspondingly, the present invention application provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the Chinese vocabulary disambiguation method when executing the computer program.

[0154] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal, and various interfaces and lines are used to connect various parts of the entire terminal.

[0155] The memory can be used to store the computer program, and the processor realizes various functions of the terminal by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0156] Accordingly, the present invention application provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the Chinese vocabulary disambiguation method.

[0157] Wherein, if the module integrated in the Chinese vocabulary disambiguation device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0158] Compared with the prior art, the present invention has the following beneficial effects:

[0159] The present invention application provides a Chinese vocabulary disambiguation method, apparatus, terminal device and storage medium, the Chinese vocabulary disambiguation method comprising: obtaining a target Chinese vocabulary, obtaining a long interpretation word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, the long interpretation word vector expression being determined according to a preset mapping relationship, the preset mapping relationship mapping at least one foreign language to the same word vector space; respectively calculating the similarity between the polysemous word vector expression and each long interpretation word vector expression, obtaining a target interpretation of the target Chinese vocabulary in the target sentence according to the long interpretation word vector expression with the highest similarity; and disambiguating the target Chinese vocabulary in the target sentence according to the target interpretation. The present invention application determines the long interpretation word vector expression according to a preset mapping relationship, and the mapping relationship maps at least one foreign language to the same word vector space. Compared with the technical solutions of the prior art that simply rely on the Chinese knowledge base, the Chinese knowledge base can be enhanced through the foreign language knowledge base, thereby improving its richness and refinement, making up for the lack of Chinese knowledge base resources, and thus improving the accuracy of the interpretation; in addition, the present invention application obtains the long interpretation word vector expression and the polysemous word vector expression of the target Chinese vocabulary in the target sentence through a cross-language model, thereby obtaining the long interpretation word vector expression with the highest similarity. Compared with the method of obtaining short interpretations in the prior art, the accuracy of the polysemous word interpretation can be further improved.

[0160] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A Chinese vocabulary disambiguation method, characterized in that: include: Acquire a target Chinese vocabulary, and obtain a long interpretation word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, wherein the long interpretation word vector expression is determined according to a preset mapping relationship, and the preset mapping relationship maps at least one foreign language to the same word vector space; respectively calculating the similarity between the word vector expression of the polysemous word and the word vector expression of each long interpretation word, and obtaining the target interpretation of the target Chinese word in the target sentence according to the long interpretation word vector expression with the highest similarity; and disambiguating the target Chinese word in the target sentence according to the target interpretation; The step of obtaining, based on the target Chinese vocabulary and the pre-trained cross-language model, a long-meaning word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence comprises: Acquire a plurality of foreign language corpora corresponding to the target Chinese vocabulary; the foreign language corpora contain a plurality of definitions; Inputting the foreign language corpus into the cross-language model; Based on the output of the cross-language model, a first label and a plurality of second labels corresponding to the target Chinese vocabulary are obtained, wherein the first label is a label corresponding to the target interpretation, and the second label is a hidden layer label of the position of the target Chinese vocabulary in the target sentence; Filter out the vectors of the foreign language corpus corresponding to the first label and the vectors of the foreign language corpus corresponding to the second label; perform a first average processing on the vectors of the foreign language corpus corresponding to the filtered first label, and obtain the vector expression of the long interpretation word based on the first average processing result; perform a second average processing on the vectors of the foreign language corpus corresponding to each second label, and obtain the polysemous word vector expression of the target Chinese vocabulary in the target sentence based on the second average processing result; The preset mapping relationship includes a first mapping relationship or a second mapping relationship; before obtaining the long-meaning word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence, it also includes: Acquire multiple synonym sets and multiple concept word sets corresponding to the target Chinese vocabulary, wherein the synonym sets include foreign language synonyms and Chinese synonyms, and the concept word sets include foreign language words and Chinese words, the synonym sets are acquired from a foreign language knowledge base, and the concept word sets are acquired from a Chinese knowledge base; When a foreign language word is consistent with a foreign language synonym, or when a Chinese word is consistent with a Chinese synonym, a first mapping relationship is constructed between the corresponding synonym set and the corresponding concept word set; otherwise, the overlap between each synonym set and each concept word set is obtained, and a second mapping relationship is constructed between the synonym set with the highest overlap and the concept word set.

2. A Chinese vocabulary disambiguation method as claimed in claim 1, characterized in that: Before obtaining the first label and the plurality of second labels corresponding to the target Chinese vocabulary, the method further includes: Performing word segmentation processing on the foreign language corpus to obtain a word list; Initialize the corresponding word vector according to the word list; Analyzing context information of each word in the word list using word co-occurrence information in a preset corpus, wherein the context information includes co-occurrence relationships of words in the context; Based on the co-occurrence relationships, a co-occurrence matrix between words is obtained; and the co-occurrence matrix is ​​decomposed to obtain an initial word vector representation; The initial word vector representation is subjected to multi-level clustering; the part of speech and dependency relationship of each word in the word list are fused with the clustering result to obtain an enhanced word vector representation of the foreign language corpus.

3. A Chinese vocabulary disambiguation method as claimed in claim 1, characterized in that: The calculation formula of the overlap degree is: ； in, is the overlap between A and B, A is the synonym set, and B is the concept word set.

4. A Chinese vocabulary disambiguation method as claimed in claim 1, characterized in that: The obtaining of the overlap between each synonym set and each concept word set includes: Get the number of words in the synonym set and the number of words in the concept set; According to the number of words in the synonym set and the number of words in the concept word set, the number of overlapping words is determined; According to the number of overlapping words, the overlapping ratio is calculated; The overlap ratio includes a first ratio of the overlapped words in the synonym set and a second ratio of the overlapped words in the concept word set; When both the first proportion and the second proportion are greater than a preset ratio value, determining the proportion of each part of speech in the synonym set and the proportion of each part of speech in the concept word set according to the corpus corresponding to the synonym set and the corpus corresponding to the concept word set; The synonym set is sorted according to the proportion from large to small to obtain a first sorting result; the concept word set is sorted according to the proportion from large to small to obtain a second sorting result; Verifying the first sorting result and the second sorting result; when the two are consistent, verifying the overlapping words through a preset concept tree structure; When the verification is passed, the degree of overlap between the synonym set and the concept word set is determined according to the overlap ratio.

5. A Chinese vocabulary disambiguation method as claimed in claim 1, characterized in that: The respectively calculating the similarity between the polysemous word vector expression and each long interpretation word vector expression includes: The similarity is calculated according to the following formula: ； Among them, Sim(D i ,D j ) is the long meaning word vector D i And the polysemous word vector D j The similarity between them, n refers to the dimension of the quantity, t ik is the long meaning word vector D i The kth number in t jk D is the polysemous word vector j The kth number in .

6. A Chinese vocabulary disambiguation device, characterized in that: It includes a first determination module and an interpretation determination module; wherein, The first determination module is used to obtain a target Chinese vocabulary, and obtain a long interpretation word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, wherein the long interpretation word vector expression is determined according to a preset mapping relationship, and the preset mapping relationship maps at least one foreign language to the same word vector space; The interpretation determination module is used to respectively calculate the similarity between the polysemous word vector expression and each long interpretation word vector expression, and obtain the target interpretation of the target Chinese vocabulary in the target sentence according to the long interpretation word vector expression with the highest similarity; and disambiguate the target Chinese vocabulary in the target sentence according to the target interpretation; The first determination module obtains a long-meaning word vector expression of the target Chinese vocabulary and a polysemous word vector expression of the target Chinese vocabulary in a target sentence according to the target Chinese vocabulary and a pre-trained cross-language model, including: The first determination module obtains a number of foreign language corpora corresponding to the target Chinese vocabulary; the foreign language corpora contain a plurality of definitions; Inputting the foreign language corpus into the cross-language model; Based on the output of the cross-language model, a first label and a plurality of second labels corresponding to the target Chinese vocabulary are obtained, wherein the first label is a label corresponding to the target interpretation, and the second label is a hidden layer label of the position of the target Chinese vocabulary in the target sentence; Filter out the vectors of the foreign language corpus corresponding to the first label and the vectors of the foreign language corpus corresponding to the second label; perform a first average processing on the vectors of the foreign language corpus corresponding to the filtered first label, and obtain the vector expression of the long interpretation word based on the first average processing result; perform a second average processing on the vectors of the foreign language corpus corresponding to each second label, and obtain the polysemous word vector expression of the target Chinese vocabulary in the target sentence based on the second average processing result; The preset mapping relationship includes a first mapping relationship or a second mapping relationship; the Chinese vocabulary disambiguation device also includes a mapping configuration module; the mapping configuration module is used for: before the first determination module obtains the long interpretation word vector expression of the target Chinese vocabulary and the polysemous word vector expression of the target Chinese vocabulary in the target sentence: Acquire multiple synonym sets and multiple concept word sets corresponding to the target Chinese vocabulary, wherein the synonym sets include foreign language synonyms and Chinese synonyms, and the concept word sets include foreign language words and Chinese words, the synonym sets are acquired from a foreign language knowledge base, and the concept word sets are acquired from a Chinese knowledge base; When a foreign language word is consistent with a foreign language synonym, or when a Chinese word is consistent with a Chinese synonym, a first mapping relationship is constructed between the corresponding synonym set and the corresponding concept word set; otherwise, the overlap between each synonym set and each concept word set is obtained, and a second mapping relationship is constructed between the synonym set with the highest overlap and the concept word set.

7. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the Chinese vocabulary disambiguation method as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the Chinese vocabulary disambiguation method according to any one of claims 1 to 5.

Images