Word distribution and document feature based automatic classification method for spam comments

Abstract

The invention discloses a word distribution and document feature based automatic classification method for spam comments. The method comprises: firstly, collecting network comments and performing word segmentation on the comments to obtain a keyword set; secondly, establishing a word distribution matrix, training a language model, and calculating a classification probability of unlabeled network comments belonging to normal comments or the spam comments; thirdly, extracting document features of the network comments, and calculating the classification probability of the unlabeled network comments; and finally, calculating a weighted average of the classification probabilities, and repeating the steps until the classification probabilities calculated for successive two times are same or a given number of iterations is reached. The method comprehensively considers word distribution features and the document features in the network comments, automatically finishes network comment classification through a self-learning policy, and assists in identification of the spam comments in the network comments. The method is simple in calculation and high in universality and expansibility, can carry out real-time classification on a great amount of comments by only a small amount of network comments with labels, and meets an application demand of quickly identifying the spam comments in the instantly updated network comments.

Description

technical field [0001] The invention relates to the field of computer applications, in particular to an automatic classification method for assisting in identifying rubbish comments in massive Internet comments. technical background [0002] With the rapid development of Internet technology, a variety of emerging network communication and communication methods have been promoted. Internet users can post various comments conveniently and quickly. With its good freedom, real-time and convenience, network communication is gradually changing the way people communicate. [0003] The development of network technology has two sides. The freedom of users to post comments and the powerful dissemination ability of the Internet are often used by some users to post various commercial advertisements or malicious information into online comments. In recent years, the spread of spam comments on the Internet has intensified, and various commercial advertisements based on spam comments and...

AI Technical Summary

Problems solved by technology

At present, there are still limitations in the classification research on the automatic identification of spam comments in the network: 1) the scalability is not strong, most classification methods can only target specific application scenarios, and it is difficult to expand; 2) the extracted features are single, and the existing The current classification method only measures the similarity of comments, and does not comprehensively consider word distribution characteristics and document features; 3) It is highly dependent on the data set and requires a large number of comment labels; it cannot meet the needs of automatic classification of spam comments in the Internet

Images