Social-media short text filtering method based on structure and text information

Abstract

The invention discloses a social-media short text filtering method based on structure and text information. The method includes the following steps that 1, the structural characteristics of a short text are judged, and junk information is deleted; 2, the core of the text is extracted, a judge structure judges whether a retained segment text contains the core information of a described event, if nocore information exists, the information is determined as junk information, and if the core information exists, core components are extracted; 3, textual features are extracted, and the core components of the text obtained in the step 2 are mapped to a characteristic space. By scanning a participle set of the text, such structural characteristics whether junk information exists or not can be judged, and mass data in the social network is thus easily and efficiently processed; by identifying characteristics of words, sentence patterns and the like, the feature selection purpose can be achieved, based on the method in which word2vec word vectors are added so as to obtain the average, a sentence vector is constructed, the calculation amount of a classifier model in the training process is reduced, and the semantic information of the text can be well represented.

Description

technical field [0001] The invention relates to a method for filtering short texts in social media based on structure and text information. Background technique [0002] With the development of technologies such as Web2.0, social media, and mobile Internet, every netizen has become a creator and disseminator of Internet information, which promotes the explosive growth of Internet text information. At the same time, the form of text content on the Internet is also constantly changing, from blogs to light blogs and microblogs, from emails to forums and instant messaging, from news to comments, etc. A notable feature is that the length of these text messages is increasing. shorter. This is because the writing of short text messages is easy and casual, and the release is more convenient. At the same time, short text messages are more concise and compact than long text messages, which can save other users' time and energy in reading messages. Short text information has much wid...

AI Technical Summary

Problems solved by technology

For example, the messages posted by individual users on social networks to record personal status not only contain a large number of abbreviations, slang and even misspelle

Images