Document similarity measurement method and system based on keyword position structure distribution

Abstract

The invention discloses a document similarity measurement method and system based on keyword position structure distribution, which comprises the following steps of storing two documents W1 and W2 each of which is provided with a plurality of natural segments; setting any target keyword set, searching all paragraph numbers and the position information of each keyword in the document W1 and the document W2, and marking the paragraph numbers and the position information by adopting a triple; generating the position distribution sequences of the keywords in the documents W1 and W2 according to the paragraph numbers and the position information; and calculating the similarity of the position distribution sequences of the keywords in the documents W1 and W2 according to the position distribution sequences of the keywords in the documents W1 and W2, thereby obtaining the weighted similarity of the two documents. According to the document similarity measurement method, the deviation of the document word and sentence semantic angle measurement similarity can be avoided, the defect that when the similarity is measured from the angle of keywords in an existing method, the features of the keywords are extracted from the document full-text distribution structure can be overcome, the practicability is higher, and the accuracy is higher.

Description

technical field [0001] The present invention relates to the technical field of document similarity measurement, in particular to a document similarity measurement method and system based on keyword position structure distribution. Background technique [0002] The similarity analysis and calculation between documents has a wide range of applications in information retrieval, data mining, machine translation, document duplication detection and other fields. A brief introduction to common document similarity calculation methods is as follows: cosine similarity, which converts documents into vector models based on keywords, and measures them by calculating the cosine similarity of documents; simple shared lexical method, which calculates the total number of characters of words shared by two documents Divide by the longest document character count to evaluate document similarity. Edit distance, also known as Levenshtein distance, is measured by the minimum number of editing ope...

AI Technical Summary

Problems solved by technology

[0003] Purpose of the invention: In order to overcome the deficiencies of the prior art, the present invention provides a method for measuring document similarity based on keyword position structure distribution, which can solve the problem of document words and sentences The problem of deviation in measuring similarity from a semantic perspective; it can also avoid the problem of insufficient extraction of keywords in the full-text distribution structure feature of the document when existing methods measure similarity from the perspective of keywords. The present invention also provides a method based on keyword positions A Document Similarity Metric System Based on Structural Distribution

Images