Text similarity calculation method based on TF-IDF and word vectors

Abstract

The invention relates to a TF-IDF and word vector-based text similarity calculation method. The method comprises the following steps of: 1, acquiring a first text and a second text to be compared, and performing word segmentation processing to form a first word set and a second word set; 2, calculating the TF-IDF value of each word in the word set by using a preset TF-IDF model, and solving a word vector by using a preset word vector model; 3, finding out a public word set and left and right adjacent word sets related to each public word through a semantic matching algorithm; and 4, inputting the TF-IDF values of the common words and the adjacent words of the common words into a first similarity model to calculate a first similarity, inputting the TF-IDF values of the word set, the word vectors, the common words and the like into a second similarity model to obtain a second similarity, and finally inputting the TF-IDF values of the word set, the word vectors, the common words and the like into a fusion model to obtain a text similarity. According to the method, multi-dimensional information is integrated, the text similarity degree is accurately measured, and the text processing efficiency and precision are effectively improved.

Claims

1. A method for calculating text similarity based on TF-IDF and word vectors, characterized in that: The calculation method includes: S1, obtaining a first text and a second text to be compared, and performing word segmentation on the first text and the second text to obtain a first word set consisting of the word segmentation results of the first text and a second word set consisting of the word segmentation results of the second text; S2, calculating the TF-IDF value of each word in the first word set and the second word set based on a preset TF-IDF model, and calculating the word vector of each word in the first word set and the second word set using a preset word vector model; S3, obtaining a common word set from the first word set and the second word set by a semantic matching algorithm, and extracting from the first word set and the second word set the left adjacent word set, the right adjacent word set, the common left adjacent word set, and the common right adjacent word set of each common word in the common word set in the first text and the second text, respectively; S4, outputting the TF-IDF value of each common word in the common word set, the TF-IDF value of each left-adjacent word in the left-adjacent word set, and the TF-IDF value of each right-adjacent word in the right-adjacent word set to a preset first similarity model for calculation to obtain a first similarity between the first text and the second text; S5, inputting the TF-IDF value of each word in the first word set and the second word set, the word vector of each word in the first word set and the second word set, the common word, the left adjacent word, and the right adjacent word into a preset second similarity model for calculation to obtain a second similarity between the first text and the second text; S6: Input the first similarity and the second similarity into a preset fusion model for calculation to obtain the text similarity between the first text and the second text.

2. A text similarity calculation method based on TF-IDF and word vector according to claim 1, characterized in that, Performing word segmentation processing on the first text and the second text specifically includes: S11, performing word segmentation processing on the first text and the second text respectively, so as to split the first text and the second text into individual words, and during the splitting process, monitoring a plurality of sequentially adjacent words in real time and simultaneously performing matching determination work; S12, when the matching determination work determines that a word string formed by connecting a plurality of sequentially adjacent words matches an entry in the synonym database, a merge connection operation is immediately performed to merge the sequentially adjacent words into a single word; S13, repeating the operation flow of step S11 to step S12 in a loop until the traversal of the first text and the second text is completed, and a first word segmentation result corresponding to the first text and a second word segmentation result corresponding to the second text are obtained.

3. The method for calculating text similarity based on TF-IDF and word vector according to claim 2, characterized in that: The method for constructing the TF-IDF model specifically includes: S21, obtaining a first text training corpus, and performing word segmentation processing on the first text training corpus to obtain a first word sequence; S22, according to the TF-IDF calculation formula, counting the term frequency TF of each word in the first word sequence in a single document and the inverse document frequency IDF of each word in the first word sequence in the entire document set; S23, according to the word frequency TF of each word in the first word sequence in a single document and the inverse document frequency IDF of each word in the first word sequence in the entire document collection as feature data, input into the first preset model for training to obtain a TF-IDF model.

4. The method for calculating text similarity based on TF-IDF and word vector according to claim 3, wherein: The method for constructing the word vector model specifically includes: S24, obtaining a second text training corpus, and performing word segmentation processing on the second text training corpus to obtain a second word sequence; S25: Input the second word sequence as feature data into a second preset model for training to obtain a word vector model.

5. The method for calculating text similarity based on TF-IDF and word vector according to claim 4, characterized in that: Calculating a first similarity between the first text and the second text using the first similarity model specifically includes: S41, calculating the weight of the TF-IDF value of each common word in the common word set, the calculation formula is as follows: Where n is the number of non-repeated words in the first word set and the second word set; na is the number of adjacent directions and has a fixed value of 2, which is used to represent left adjacent and right adjacent; represents the TF-IDF value of each left-adjacent word in the left-adjacent word set and the TF-IDF value of each right-adjacent word in the right-adjacent word set; Represents the TF-IDF value of each word in the first word set and the second word set; S42, calculating a first similarity between the first text and the second text according to the weight of the TF-IDF value of each common word in the common word set, the TF-IDF value of each common word in the common word set, the TF-IDF value of each left adjacent word in the left adjacent word set, and the TF-IDF value of each right adjacent word in the right adjacent word set, using the following calculation formula: Among them, wco t,i The weight of the TF-IDF value of each common word in the common word set; represents the TF-IDF value of each word in the first word set and the second word set; tfidf t,i The TF-IDF value of each common word in the common word set; SIM1(T1, T2) represents the first similarity between the first text and the second text; nt is the number of texts to be compared and is fixed at 2; m is the number of common words in the common word set, and n is the number of non-repeated words in the first word set and the second word set.

6. The method for calculating text similarity based on TF-IDF and word vector according to claim 5, characterized in that: Calculating the second similarity between the first text and the second text using the second similarity model specifically includes: S51, respectively calculating the word vector of each word in the first word set and the second word set, using the following formula: w word =0,if word∈C s if word∈C s and adjacency∈(AdjLeft∪AdjRight) Wherein, Cs represents the common word set in the first word set or the second word set; nw represents the number of words in the first word set or the second word set; tfidf word represents the TF-IDF value of each common word in the common word set in the first word set or the second word set; word represents the vector weight of each word in the first word set or the second word set; AdjLeft represents the left adjacent word set of a common word in the common word set in the first word set or the second word set; AdjRight represents the right adjacent word set of a common word in the common word set in the first word set or the second word set; wco word represents the weight of each common word in the common word set in the first word set or the second word set; adjacency representing a vector weight of adjacent words of each common word in the common word set in the first word set or the second word set; S52: Calculate the sentence vector of the first text and the sentence vector of the second text according to the vector of each word in the first word set and the second word set, and the calculation formula is: Where nl represents the number of words in the first word set or the second word set, w word V represents the vector weight of each word in the first word set or the second word set; word A vector representing each word in the first word set or the second word set; V t a sentence vector representing the first text or a sentence vector representing the second text; S53: Calculate the cosine distance between the sentence vector of the first text and the sentence vector of the second text, and use the cosine distance as the second similarity between the first text and the second text. The calculation formula of the cosine distance is: Wherein, SIM2(T1, T2) represents the second similarity between the first text and the second text; V t1 V represents the sentence vector of the first text; t2 Represent the sentence vector of the second text.

7. The method for calculating text similarity based on TF-IDF and word vector according to claim 6, characterized in that: The text similarity between the first text and the second text is calculated using the fusion model, and the calculation formula is as follows: SIM(T1,T2)=λ*sign(m)*SIM1(T1,T2)+(1-λ*sign(m))*SIM2(T1,T2) Wherein, λ is the adjustment factor; sign(m) is the sign function, m is the number of common words in the common word set; SIM1(T1, T2) represents the first similarity between the first text and the second text; SIM2(T1, T2) represents the second similarity between the first text and the second text.

Images