Method for determining document similarity based on improved Jaccard coefficients

Abstract

The invention discloses a method for determining document similarity based on improved Jaccard coefficients. The method comprises: step 1, respectively determining the corresponding number (which is shown in the description) of each element wi, the length of which is K in a document X, and the corresponding number (which is shown in the description) of each element wj, the length of which is K in a document Y; step 2, calculating the proportion (which is shown in the description) of each element wi in the document X; step 3, calculating the proportion (which is shown in the description) of each element wj in the document Y; step 4, calculating the Jaccard similarity (which is shown in the description) of a common element wh in the document X and the document Y; step 5, calculating the Epsilon (wh) of the element wh in elements, the n-Gram length of which is K in the document X and the document Y; step 6, calculating a parameter F(wh) representing whether the element wh simultaneously exists in the document X and the document Y; and step 7, setting a symbol (which is shown in the description) as the similarity of the document X and the document Y. According to the method for determining the document similarity based on the improved Jaccard coefficients, by considering the proportion of each element and sample in the documents and the contribution degree to the similarity of multiple documents, a problem that inter-document similarity calculation is inexact in the prior art is effectively solved.

Description

Technical field [0001] The invention relates to the field of information retrieval, and in particular to a method for determining document similarity based on an improved Jaccard coefficient. Background technique [0002] With the rapid development of modern computer technology and the rapid popularization of the Internet, online data resources are also rapidly increasing. This rich data resource provides convenience for people's lives and improves people's work efficiency. While these data resources provide convenience to people, many problems have also appeared, such as plagiarism of academic papers and news reprints. In this context, duplicate detection has emerged. Similarity calculation has a wide range of application prospects. Currently, it is mainly used in academic paper duplicate checking, electronic file copyright, text clustering, questionnaire survey sorting, search engine deduplication, etc. [0003] The amount of detection data for similarity data is huge. On Baid...

AI Technical Summary

Problems solved by technology

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art, to provide a method for determining document similarity based on the improved Jaccard coefficient, by considering the weight of each element, sample in the document and its contribution to the similarity of multiple documents, Effectively solve the problem of inaccurate calculation of similarity between documents existing in the prior art

Images