A method and device for calculating word meaning similarity based on adjacent word features

Abstract

This invention provides a method for calculating semantic similarity based on adjacent word features, relating to the field of natural language processing. First, in the example word extraction module, example words are extracted from the example sentence and the sentence to be matched using the longest common substring algorithm. Second, in the example sentence processing module, word segmentation is performed on the example sentence excluding the example words, and part-of-speech tagging is applied to the example words. Then, in the feature extraction module, features are extracted from the words surrounding the example words in the example sentence. The correlation between the surrounding words and the example words and option words is calculated using a corpus. The calculated results are weighted and combined with their respective features to form the features of the example words and option words. Finally, in the option processing module, the part-of-speech of the option words is compared with that of the example words, and a similarity score is calculated based on the features of the option words to select the optimal option word. This invention determines the features of the current word by extracting features from surrounding words and calculating correlation; it uses a corpus to calculate the mutual information between two words to achieve the correlation between the two words.

Description

Technical Field

[0001] This invention relates to a method and apparatus for calculating semantic similarity based on adjacent word features, specifically a method for calculating semantic similarity of words, belonging to the field of natural language processing. Background Technology

[0002] Semantic similarity of words plays an irreplaceable role in the field of natural language processing. However, the semantic relationships between words are very complex, and it is difficult to measure the degree of similarity between words using a simple numerical value. The same pair of words may seem very similar in one aspect, but may be quite different from another perspective. Semantic similarity calculation has wide applications in many fields, such as information retrieval, information extraction, text classification, word sense disambiguation, and instance-based machine translation. Currently, there are two common methods for calculating semantic similarity: one is based on a thesaurus, where all words are organized in one or several tree structures, and the path length between two nodes can be used as the semantic distance; the other is based on large-scale corpus statistics, which uses the probability distribution of word context information to calculate semantic similarity between words. The first method relies entirely on a semantic dictionary when calculating semantic similarity, and the calculation results are very sensitive to the quality and completeness of the semantic dictionary; the second method requires a large-scale corpus for statistical analysis, and a small corpus may lead to inaccurate calculation results. This invention introduces part-of-speech discrimination to calculate word semantic similarity based on corpus statistics, thus obtaining accurate results even if the corpus is not large enough. Summary of the Invention

[0003] To address the shortcomings of existing word sense similarity calculation methods, the present invention aims to provide a method for calculating word sense similarity by replacing the feature vector of the word to be matched with the feature vector of adjacent words.

[0004] The technical solution implemented in this invention is: a vocabulary matching algorithm based on adjacent word features. The core of this algorithm is to replace the features of example words and option words with the feature weighting of the words surrounding the example word, and then calculate the similarity score between the example word and the option word to select the optimal option word. The specific steps of the method are as follows:

[0005] Step 1: Extract example words from the example sentence and the sentence to be matched using the maximum common substring algorithm.

[0006] The maximum common substring algorithm is used to obtain the common substring array between the example sentence and the sentence to be matched. The color tags are removed from the common substrings in the example sentence containing the common substring array, and the remaining characters with color tags are the example words. The example sentence and the sentence to be matched have the same length, and except for the tags and example words, the other words and punctuation marks are exactly the same.

[0007] Step 1.1 Obtain the example sentence text1 and the sentence to be matched text2, where text1 = {a1, a2, ..., a...} n},a1,a2,...,a n Let each character in text1 represent a character in text2, and text2 = {b1, b2, ..., [mask], ..., b}. n},b1,b2,...,b n Each of the n characters in text2 represents a different character.

[0008] Step 1.2 Perform example word recognition on example sentence text1 and sentence text2 to be matched;

[0009] Let A = text1 and B = text2, and label text1 with a color. Use dynamic programming to find the longest common substring between A and B. The specific method is as follows:

[0010] Define the state transition equation as follows:

[0011]

[0012] The length of the longest common substring of A and B can be calculated using the above formula as max(c[i,j]), where i,j∈{1,...,n}. After obtaining the length of the longest common substring, marking its initial position will yield the entire longest common substring. The calculated longest common substring of A and B is then stored in an array L, where L={l1,...,l p}, p≤n; Based on the longest common substring array L, remove the color tags of the common substrings in text1, and the remaining characters with color tags are the example words c. f .

[0013] Step 2 involves segmenting the characters in the example sentences (excluding the example words) into words and tagging the example words with their parts of speech.

[0014] The Hidden Markov Model (HMM) is used to segment characters except for example words with color labels, and the Viterbi algorithm is used to perform part-of-speech tagging on example words with color labels.

[0015] Except for example word 'c', extract text1 from text1. f The characters are segmented using the Hidden Markov Model (HMM) to obtain the segmented text C = {c1,...,c...} f ,...,c t}, t≤n-1, then use the Viterbi algorithm to process the example word c f Part-of-speech tagging is performed to obtain its part-of-speech Z. f .

[0016] The following is a table of Chinese parts of speech:

[0017]

[0018]

[0019] Step 3: Extract feature vectors of example words and option words from the segmented example sentences.

[0020] First, the word segments surrounding the example words marked with color are converted into word vectors for feature extraction. Then, the mutual information is used in the corpus to calculate the correlation between the example words and the option words and the surrounding word segments. Finally, the correlation is used to weight the word vectors of the surrounding word segments to form a new vector as the feature vector of the example words and the option words.

[0021] Step 3.1 Use the word2vec model to obtain the exception word c f The word vector matrix E = [e1,...,e2] for other word segments is... i ,...,e t The word vector matrix E is a d×t matrix, where d represents each word segmentation feature vector e. i The feature dimensions of i∈{1,...,t}.

[0022] Step 3.2 Calculate example word c in the corpus using the following formula. f and option word o j j∈{1,...,4} and example word c f Other word segmentation relevance:

[0023]

[0024] Among them, g is the example word c f and option word o j c t Except for example word c f Other word segmentations besides P(g,c) t ) represents g and c in the corpus. t The probabilities of simultaneous occurrence, P(g), P(c) t ) for g, c t The probability of independent occurrence, I(g;c) p ) for g, c t Mutual information content;

[0025] Step 3.3 After calculating example word c f , option word o j After determining the relevance to other word segments, the word vector e for each segment is calculated based on the relevance. i For example word c f , option word o j Word vector weight w (g,i)

[0026]

[0027] Step 3.4 Calculate the word vector e for each word segmentation. i Multiply by its corresponding weight w (g,i) For example, word c f , option word o j The feature vector E extracted from this word segmentation g E g Including example word c f eigenvector E f And the option word o j eigenvector E oj ;

[0028]

[0029] Step 4: Calculate the similarity score between the option and the example word to determine the optimal option word.

[0030] The similarity score of the option words is calculated by combining the part-of-speech and feature vector of the example words and the part-of-speech and feature vector of the option words. The option word with the highest similarity score is the optimal option word.

[0031] Step 4.1 Compare the parts of speech of example words Z f Part of speech Z of each option word oj , making s c From the part-of-speech comparison results, we can obtain:

[0032]

[0033] Step 4.2 Calculate example word c f With each option word o j similarity score s j :

[0034] s j =s cj ·(E f T ·E oj )

[0035] Step 4.3 Compare example words c f With each option word o j The fractions between s j The option with the highest score is the optimal option.

[0036] The present invention also provides a word semantic similarity calculation device based on adjacent word features, comprising the following modules:

[0037] The example word extraction module is used to extract example words from the example sentence and the sentence to be matched using the maximum common substring algorithm;

[0038] The example sentence processing module is used to segment words in example sentences (excluding example words) and to tag the parts of speech of example words.

[0039] The feature extraction module is used to extract feature arrays from example words and option words;

[0040] The option processing module is used to calculate the similarity score between the option and the example word, thereby obtaining the optimal option word.

[0041] The beneficial effects of this invention are: this invention determines the features of the current word by extracting the features of surrounding words and calculating the relevance; it uses a corpus to calculate the mutual information of two words to calculate the relevance of the two words; compared with word sense similarity calculation based on large-scale corpora, the part-of-speech discrimination introduced in this invention enables accurate calculation results even without a large-scale corpus. Attached Figure Description

[0042] Figure 1 This is a flowchart of a word sense similarity calculation method based on adjacent word features proposed in this invention.

[0043] Figure 2 This is a flowchart of the example word extraction module of a word meaning similarity calculation method based on adjacent word features proposed in this invention.

[0044] Figure 3 This is a flowchart of the example sentence processing module of a word sense similarity calculation method based on adjacent word features proposed in this invention.

[0045] Figure 4 This is a flowchart of the feature extraction module of a word sense similarity calculation method based on adjacent word features proposed in this invention.

[0046] Figure 5 This is a flowchart of the option processing module of a word sense similarity calculation method based on adjacent word features proposed in this invention. Detailed Implementation

[0047] The technical solutions and embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The following embodiments are only for illustrating the technical solutions of the present invention and are merely examples; they should not be used to limit the scope of protection of the present invention. The technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art.

[0048] like Figure 1As shown, the present invention provides an apparatus for calculating semantic similarity based on adjacent word features, including an example word extraction module, an example sentence processing module, a feature extraction module, and an option processing module.

[0049] The specific methods for calculating word sense similarity using the above-mentioned device include:

[0050] Step 1: Extract example words from the example sentence and the sentence to be matched using the maximum common substring algorithm;

[0051] Step 2: Segment the characters in the example sentences except for the example words, and tag the example words with their parts of speech;

[0052] Step 3: Extract feature vectors of example words and option words from the example sentences after word segmentation;

[0053] Step 4: Calculate the similarity score between the option and the example word to obtain the optimal option word.

[0054] This embodiment illustrates a specific word selection process. The problem consists of an example sentence and a sentence to be matched containing the words to be matched. The words to be matched are marked with tags, and the lengths of the example sentence and the sentence to be matched are kept consistent. The options are four words that have already been marked with their parts of speech.

[0055] Example sentence text1, sentence to be matched text2, option word o j And the parts of speech are as follows:

[0056]

[0057] like Figure 2 As shown, the example word c is extracted using the example word processing module by combining the example sentence text1 and the sentence to be matched text2. f First, add color tags to the example sentences and the sentences to be matched:

[0058] Example sentences text1 I use a bowl to eat food. Sentence to be matched text2 I use [mask] to eat food.

[0059] The longest common substring algorithm is used to obtain the longest common substring array L = {"I use", "eat food"}. Based on array L, the color tags of the common substrings between example sentence text1 and the sentence to be matched text2 are removed. The characters with color tags remaining in the example sentence are the example word c. f :

[0060] Example sentences text1 I use a bowl to eat food. Sentence to be matched text2 I use \color{[mask]} to eat food. Example words <![CDATA[c f ]]> bowl

[0061] Example word c f Then remove the color tag from the example sentence text1.

[0062] like Figure 3As shown, in the example sentence processing module, HMM is used to perform word segmentation on the example sentence text1, resulting in the segmented example sentence text1 and the example word c excluding the example word c. f Other word segmentations c t :

[0063]

[0064] The Viterbi algorithm was used to analyze example word c. f Perform part-of-speech tagging on the example sentence text1 to obtain the part-of-speech tag Z of the example word. f for:

[0065] Example words <![CDATA[c f ]]> bowl Example word part of speech <![CDATA[Z f ]]> n

[0066] like Figure 4 As shown, the word2vec model is used in the feature extraction module to extract word segmentation c. t eigenvector e i Each word segmentation feature vector is 10-dimensional, and the pre-training data comes from Chinese Wikipedia. The extracted feature vectors are combined into a 10×4 feature vector matrix E:

[0067]

[0068] The CCL corpus used in this invention has an estimated total size of 6,256,178 sentences. The corpus was used to statistically analyze the frequency of example words, each option word, each segmentation word, the frequency of example words and each segmentation word appearing together, and the frequency of each option word and each segmentation word appearing together.

[0069]

[0070]

[0071]

[0072]

[0073] The correlation degree I(c) between the example word and each word segment is calculated using the following formula. f ,c t The degree of correlation between each option word and each segmentation word I(o) i ,c t ):

[0074]

[0075] The correlation coefficient calculation results are as follows (when the calculation result exceeds six decimal places or the calculation result is negative, the correlation coefficient is taken as 0.000000):

[0076]

[0077]

[0078] The word vector e of each segmentation word is calculated using the following formula. i For example word c f , option word o j Word vector weight w (g,i) :

[0079]

[0080] The weight calculation results are as follows (no calculation is performed for correlations of 0.000000):

[0081]

[0082] Since the word vector for the word "use" has a weight of 0 relative to the word vector for the example word "bowl," the word vectors for both the example word and each option word also have a weight of 0 for that word. The calculated word vectors (each with a dimension of 30×1) for the example word and each option word are as follows:

[0083] Example word vector E for "bowl" f :

[0084]

[0085] Option word "chopsticks" word vector E o1 :

[0086]

[0087] Word vector E for the option word "mouse" o2 :

[0088]

[0089] Word vector E for the option word "running" o3 :

[0090]

[0091] The word vector E for the option word "snow white" o4 :

[0092]

[0093] like Figure 5 As shown, in the option processing module, the part-of-speech Z of the example words is compared according to the following formula. f The part of speech of each option Z oj Let the comparison result be s. cj .

[0094]

[0095] The part-of-speech tagging results are as follows:

[0096]

[0097] The similarity score s between the example word and each option word is calculated using the following formula. j :

[0098] s j =s cj ·(E f T ·E oj )

[0099] The similarity score is calculated as follows:

[0100] The similarity score s1 between the example word "bowl" and the option word "chopsticks":

[0101]

[0102] The similarity score s2 between the example word "bowl" and the option word "mouse":

[0103]

[0104] The similarity score s3 between the example word "bowl" and the option word "running":

[0105] s3 = s c3 ·(E f T ·E o3 )=0·{[0.12e1 0.86e3 0.02e4]·[0.45e1; 0.49e3; 0.06e4] T =0.000000 The similarity score s3 between the example word "bowl" and the option word "snow-white":

[0106] s4 = s c4 ·(E f T ·E o4 )=0·{[0.12e1 0.86e3 0.02e4]·[0.49e1; 0.50e3; 0.01e4] T The similarity scores from highest to lowest are: s1 > s2 > s3 = s4. Therefore, "chopsticks" is the optimal option.

Claims

1. A method for calculating semantic similarity based on adjacent word features, characterized in that, Includes the following steps: Step 1: Extract example words from the example sentence and the sentence to be matched using the maximum common substring algorithm; The maximum common substring algorithm is used to obtain the common substring array between the example sentence and the sentence to be matched. The color tags are removed from the common substrings in the example sentence containing the common substring array, and the remaining characters with color tags are the example words. Step 2: Segment the characters in the example sentences except for the example words, and tag the example words with their parts of speech; The Hidden Markov Model (HMM) is used to segment characters except for example words with color labels, and the Viterbi algorithm is used to perform part-of-speech tagging on example words with color labels. Step 3: Extract feature vectors of example words and option words from the example sentences after word segmentation; First, the word segments surrounding the example words marked with color are converted into word vectors for feature extraction. Then, the mutual information is used in the corpus to calculate the correlation between the example words and the option words and the surrounding word segments. Finally, the correlation is used to weight the word vectors of the surrounding word segments to form a new vector as the feature vector of the example words and the option words. Step 4: Calculate the similarity score between the option and the example word to determine the optimal option word; The similarity score of the option words is calculated by combining the part-of-speech and feature vector of the example words and the part-of-speech and feature vector of the option words. The option word with the highest similarity score is the optimal option word. The specific steps of Step 3 are as follows: Step 3.1 Use the word2vec model to obtain the exception words. Other word vector matrices ; Step 3.2 Calculate example words in the corpus using the following formula. and option words , and example words Other word segmentation relevance: ； in Example words and option words , Except for example words Other word segmentations, For the corpus , The probability of them occurring simultaneously , for , The probability of appearing independently for , Mutual information content; Step 3.3 After calculating the example words Option words After determining the relevance to other word segments, the word vector for each segment is calculated based on the relevance. For example words Option words Word vector weights ； Step 3.4 Calculate the word vector for each segmented word. Multiply by its corresponding weight That is, example words Option words Feature vector extracted from this word segmentation , Including example words eigenvectors and option words eigenvectors ; 。 2. The word sense similarity calculation method based on adjacent word features as described in claim 1, characterized in that, The example sentences and the sentences to be matched are of the same length, and except for the markers and example words, the other words and punctuation marks are exactly the same.

3. The word sense similarity calculation method based on adjacent word features as described in claim 1, characterized in that, The specific steps of Step 1 are as follows: Step 1.1 Obtain example sentences matching sentence , , They represent the contents of text1 respectively. n One character, , They represent the text in text2 respectively n One character; Step 1.2 Example Sentences and the sentence to be matched Perform example word recognition; make , and give Apply color labels; Solve using dynamic programming and The longest common substring in the sequence will be found by solving for the longest common substring in the sequence. and Store the longest common substring in an array middle, , Based on the longest common substring array Remove The color label of the common substring, and the remaining characters with color labels are the example words. .

4. The word sense similarity calculation method based on adjacent word features as described in claim 1, characterized in that, The specific steps of Step 2 are as follows: In addition to example words The characters are segmented using the HMM model to obtain the segmented text. , Then, the Viterbi algorithm is used to analyze the example words. Part-of-speech tagging is performed to obtain its part of speech. .

5. The word sense similarity calculation method based on adjacent word features as described in claim 1, characterized in that, The specific steps of Step 4 are as follows: Step 4.1 Compare the parts of speech of example words Part of speech of each option word ,make From the part-of-speech comparison results, we can obtain: ； Step 4.2 Calculate example words With each option word similarity score : ； Step 4.3 Compare example words With each option word The score between The option with the highest score is the optimal option.

6. An apparatus for implementing the word semantic similarity calculation method based on adjacent word features as described in claim 1, characterized in that, Includes the following modules: The example word extraction module is used to extract example words from the example sentence and the sentence to be matched using the maximum common substring algorithm; The example sentence processing module is used to segment words in example sentences (excluding example words) and to tag the parts of speech of example words. The feature extraction module is used to extract feature arrays from example words and option words; The option processing module is used to calculate the similarity score between the option word and the example word, thereby obtaining the optimal option word.

Images