Cluster-based text duplicate checking method

Abstract

The invention discloses a cluster-based text duplicate checking method. The method includes the steps: 1, for data acquisition and processing, storing text data in a database and a file server; 2, for preprocessing, subjecting the text data to word segmentation and feature vector extraction; 3, clustering the text data preprocessed in the database, and calculating center feature vectors of all class clusters; 4, for primary duplicate checking processing, extracting feature vectors of the text data, comparing the feature vectors with the center vectors of the class clusters in the database, and recording the class clusters of the center feature vectors with the distance smaller than a set threshold; 5, for secondary duplicate checking processing, comparing the feature vectors of the text data with the feature vectors of the text data in the corresponding class clusters, and recording the corresponding text data of the feature vectors with the distance smaller than a certain threshold as duplicated text data, so as to realize text data duplicate checking. By the method, unnecessary duplicated comparative work can be reduced, and text duplicate checking efficiency is improved.

Description

technical field

[0001] The invention relates to the technical field of text data analysis and mining, in particular to a clustering-based text plagiarism checking method. Background technique

[0002] In recent years, with the frequent occurrence of fraudulent incidents in academia and the increasing calls for intellectual property protection, the research on text plagiarism checking technology has gradually become a research hotspot for relevant experts and scholars. At present, some scholars at home and abroad have proposed text plagiarism checking methods, which can be mainly divided into the following categories after induction:

[0003] 1. Text plagiarism checking method based on the sememe space of HowNet.

[0004] In this method, the text is firstly segmented, and then the split words are further divided into smaller semantic units "sememes". "HowNet" is based on sememes, and uses a formalized language (similar to ontology description language) to organize sememes t...

Images