Text similarity calculation method based on x2-C

Abstract

The invention discloses a text similarity calculation method based on x2-C, and particularly relates to the field of text information processing. According to the method, a convolutional neural network CNN is used to classify a test data set; calculating an initial weight of each feature word in the detection sample according to the TF-IDF; calculating a domain correlation factor by using an x < 2>-C algorithm, calculating an initial weight by using the word position factor alpha in combination with the domain correlation factor to obtain a feature word weight, establishing a word bank by using all feature words of the detection sample, and expressing the detection sample as an initial text vector in combination with the word bank and the feature word weight; utilizing a word2vec tool tocalculate the similarity degree among the words in the word bank and form a word meaning similarity degree matrix; the initial text vector is calculated by using the matrix to obtain the text vector,and finally the text vector is calculated by using a cosine similarity algorithm to obtain the similarity between the texts, so that the association degree between the feature words and the field of the feature words, the semantic relationship between the feature words and the position information of the feature words are increased, and the accuracy of text similarity calculation is improved.

Description

technical field [0001] The invention relates to the field of text information processing, in particular to a χ-based 2 -C's text similarity calculation method. Background technique [0002] Text similarity is the calculation of the degree of semantic similarity between texts. In the era of information explosion, text similarity is used in many fields. For example: question answering system, automatic marking, plagiarism checking, etc., the traditional calculation of text similarity is based on the vector space model (VectorSpaceModel, VSM), which uses TF-IDF to calculate the weight of feature words in the text and convert the text into multidimensional The text vector of space measures the similarity between texts by calculating the similarity of text vectors. However, the TF-IDF algorithm only considers the relationship between term features and documents, and does not consider the relationship with categories, resulting in low accuracy of text similarity calculation. Th...

AI Technical Summary

Problems solved by technology

However, there are disadvantages such as complex structure of CNN model, many parameters, and long running time.

Images