Document similarity measurement method and system based on keyword sequence structure

Abstract

The invention discloses a document similarity measurement method based on a keyword sequence structure, and the method comprises the steps of storing a document W1 and a document W2, setting a keywordsequence, and searching a position set where all keywords in the keyword sequence appear in the document W1 and the document W2; generating the feature sets of the keyword sequences in the documentsW1 and W2 according to the positions where the keywords appear so as to obtain the structural feature values of the keyword sequences in the documents W1 and W2; and calculating the similarity of thedocuments W1 and W2 with respect to the keyword sequence according to the structural characteristic values of the keyword sequence in W1 and W2. The method is beneficial to avoiding the deviation of the semantic angle measurement similarity of the words and sentences of the document, can avoid the defect that the causal relationship before and after a group of keywords is ignored in document distribution structure feature extraction when the similarity is measured from the angle of the keywords in an existing method, and is stronger in practicability and higher in accuracy.

Description

technical field [0001] The invention relates to the technical field of document similarity measurement, in particular to a method and system for document similarity measurement based on a keyword sequence structure. Background technique [0002] The similarity analysis and calculation between documents has a wide range of applications in information retrieval, data mining, machine translation, document duplication detection and other fields. A brief introduction to common document similarity calculation methods is as follows: cosine similarity, which converts documents into vector models based on keywords, and measures them by calculating the cosine similarity of documents; simple shared lexical method, which calculates the total number of characters of words shared by two documents Divide by the longest document character count to evaluate document similarity. Edit distance, also known as Levenshtein distance, is measured by the minimum number of editing operations require...

AI Technical Summary

Problems solved by technology

[0003] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a method for measuring document similarity based on keyword sequence structure, which can solve the semantics of document words and sentences The problem of the deviation of angle measurement similarity; also can avoid existing method when measuring similarity from keyword angle, to the insufficient problem that keyword is extracted in document full-text distribution structural feature, the present invention also provides a kind of based on keyword sequence structure document similarity measurement system

Images