Method and device for spam filtering based on short text

Abstract

The invention discloses a method for spam filtering based on a short text. The method for spam filtering based on the short text comprises the following steps that word segmentation is conducted on the text of each email and word segmentation results are obtained; sequencing is conducted on the word segmentation results through the TF-IDF technology, so that a word segmentation list is obtained; an email fingerprint of each email is calculated according to the word segmentation results; clustering processing is conducted on the emails according to the email fingerprints, and a clustering result is obtained; spam filtering is conducted according to the clustering result. The invention further discloses a device for spam filtering based on the short text. By the adoption of the method and device for spam filtering based on the short text, word segmentation and TF-IDF technology sequencing can be conducted on the texts of the emails, and noise filtering is achieved; according to the length of the text of each email, the email fingerprint of each email is calculated through one or more BKDR hash functions, and the function of the word segmentation result can be effectively enhanced; clustering processing can be conducted on the emails through similarity comparison of the fingerprints by means of normalization processing, and therefore spam filtering is achieved.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a spam filtering method based on short text and a spam filtering device based on short text. Background technique [0002] With the wide application of the Internet, e-mail is favored by people for its fast, simple and cheap advantages, and has become an efficient mass communication medium. At the same time, a large number of useless emails poured into people's mailboxes, bringing disasters to their studies and lives. Spam is what users hate most. They waste users' time, money, and network bandwidth. At the same time, they mess up users' mailboxes. Some emails are even harmful, such as containing pornographic content or viruses. According to relevant research reports, more than 10% of emails in the world are spam every day. Therefore, it is necessary to develop an effective method for effectively intercepting and filtering garbage. [0003] At present, there are many meth...

AI Technical Summary

Problems solved by technology

However, for short text information with very short text content, use the zipf rule to extract the most important feature words, and after discarding the unimportant feature words, there will be less information left. If you directly use such little information for text Classification, the final result may even be wrong, and in severe cases, normal emails may even be classified as spam, resulting in users not reading or not reading this email in time

In addition, currently a large amount of spam is often packaged in html, and by adding a large number of invisible or different-sized fonts, it interferes with the effect of text classification. Therefore, this patent is not suitable for filtering spam with very short body content

Images