Method for solving text similarity based on Gini index

Abstract

The invention discloses a method for solving a text similarity based on the Gini index. The method comprises the following steps: performing text word segmentation processing by use of the word segmentation technology, matching with a stop word list to perform a stop word elimination operation on a vocabulary, and obtaining a series of vocabulary positions and word characteristic weighted values according to the research statistics; collecting and reducing dimensions of the text vocabulary by use of a target weight function as shown in description, combining the vocabularies with high similarity according to the semantic similarity, collecting and reducing the dimensions of above characteristic words again, and solving the inter-textual similarity by use of the similarity between the vectors. Compared with the traditional text characteristic vocabulary extracting method, the method disclosed by the invention is higher in accuracy, better in application vale, and good in data processing effect; the defects of an information gain method are overcome, the result is more suitable for the experience value, the text characteristic vocabulary high-dimensional spare problem and the problem of the synonyms and polyseme are solved, the contribute degrees of different vocabularies to the text thought are computed, and the good theory basis is provided for the subsequent text similarity and text clustering.

Description

technical field [0001] The invention relates to the technical field of semantic network, in particular to a method for calculating text similarity based on the Gini index. Background technique [0002] In Chinese information processing, the calculation of text similarity is widely used in information retrieval, machine translation, automatic question answering system, text mining and other fields. At present, most text similarity algorithms are based on the vector space model, but this method will cause the problem of high-dimensional sparsity and the importance and contribution of the characteristic vocabulary set to the text are unknown. Moreover, such algorithms do not solve the natural language problems of synonyms and polysemy in text data well. These problems interfere with the efficiency and accuracy of the text similarity algorithm, and degrade the performance of similarity calculation. In order to meet the above requirements, the present invention provides a metho...

AI Technical Summary

Problems solved by technology

[0003] Aiming at the problem of high-dimensional sparse feature vector, the problem of not considering the importance and contribution of the feature vocabulary set to the text, the problem of synonyms and polysemous words, and the problem of low efficiency and accuracy of the text similarity algorithm, the present invention provides a Gini-based A Method of Exponentially Computing Text Similarity

Images