A method of document similarity discrimination based on Fourier transform

Abstract

The invention provides a document similarity distinguishing method based on Fourier transform. The method comprises the following steps: acquiring the keyword sequence Ks of a document collection S and a corresponding keyword frequency collection Ns, as well as a keyword sequence Ks' of the detection document s' relative to the document collection S and s corresponding keyword frequency collection Ns'; calculating the weight coefficient of each of the keyword sequences Ks and Ks' as well as the weight sequence FKs of the keyword sequence Ks and the weight sequence FKs' of the keyword sequence Ks'; carrying out Fourier transform to weight sequence FKs and FKs'; calculating the threshold value Omega S of similarity distance of similarity of random document in the detection document s' and the document collection S; calculating the similarity distance D (s', si) between the documents si in the detection document s' and the document collection S, and comparing the similarity distance D with the threshold value Omega S; judging whether the detection document s' and the document collection S are similar or not. The distinguishing method of document similarity based on Fourier transform provided by the invention can not only reduce the requirement to a representing method of the document while calculating similarity, but also can reduce the complexity of calculation and improve the computational efficiency.

Description

technical field [0001] The invention belongs to the technical field of information retrieval and text mining, and in particular relates to a document similarity discrimination method based on Fourier transform. Background technique [0002] As people pay more and more attention to science and technology and social development, the academic field is gradually developing towards diversification, informationization and modernization. In this situation, people urgently need efficient, comprehensive and convenient retrieval of academic information more than ever. On the other hand, people also need to prevent academic plagiarism in order to achieve the purpose of supervising and regulating dissertations and academic journals. The key to paper retrieval and plagiarism check is the comparison and calculation of similarity of text information. Therefore, the calculation of text similarity is widely used in information retrieval, text mining and other fields. It is a very basic and ...

AI Technical Summary

Problems solved by technology

Finally, the similarity between texts is finally calculated by the inner product and cosine formula, but the biggest disadvantage of this method is that when the corpus increases, the difficulty of using vectors to represent texts will increase. At the same time, as the vector dimension increases , the complexity of calculating the similarity will increase, and the efficiency will decrease accordingly.

Images