Chinese word segmentation based text similarity identifying method and device

Abstract

An embodiment of the invention discloses a Chinese word segmentation based text similarity identifying method. The method is characterized by including: filtering unidentified and meaningless characters in texts in given coded format and obtaining preprocessed texts; segmenting words of the preprocessed texts according to a preset word segmenting mode; selecting characteristic words in words obtained from word segmentation according to preset policy; sequencing the selected characteristics words to obtain a special character string, and calculating characteristic values of the texts according to the special character string; and determining similarity of the texts by comparing the characteristic values of the texts. The embodiment of the invention further discloses a Chinese word segmentation based text similarity identifying device. By the Chinese word segmentation based text similarity identifying method and device, identifying complexity can be reduced, identifying efficiency can be improved, and higher identifying correct rate can be achieved.

Description

technical field [0001] The invention relates to the technical field of text recognition, in particular to a text similarity recognition method and device based on Chinese word segmentation. Background technique [0002] With the rapid development of Web 2.0 technology and SNS, although information dissemination is more convenient, there are more and more repeated information. According to public statistics, the average repetition rate of web pages is about 4. That is to say, when you see an article information through a link, there are an average of 3 other different links that also give the same or basically similar content. [0003] For the same or similar information, some existing websites, communities, etc. will identify the text content in web pages or articles, extract some feature information in the text, and judge the similarity between texts based on these feature information, and pass Clustering, filtering and other methods process similar texts, reduce excessive...

AI Technical Summary

Problems solved by technology

[0009] Although the above two algorithms can guarantee a high recognition accuracy and reduce the manslaughter caused by the keyword extraction method, the amount of calculation required by these two algorithms is quite large, and their time complexity (extracting feature information The time required for the comparison process with the feature information) and the complexity of the space (the storage space required for the extracted feature information) are very high, which greatly reduces the efficiency of text similarity recognition and is difficult to apply to the application of massive text business

Images