Event detection-oriented short text data filtering method for social networks

Abstract

The invention discloses an event detection oriented social network short text data filtering method, which comprises the following steps of: preprocessing the social network short text data, extracting features of the social network short text data, and training a GBDT classifier to classify the social network short text data. A method and apparatus are disclosed based on the background characteristic of a user, the syntactic features of the text and the influence features of the text are analyzed, then 20-dimensional classification features are extracted. Finally, GBDT algorithm is used to classify the short text data, filter the data classified as useless information, and retain the potentially useful value information data, so as to provide effective data input for event detection.

Description

technical field [0001] The invention belongs to the technical field of natural language processing, and in particular relates to an event detection-oriented social network short text data filtering method. Background technique [0002] With the development of technologies such as web2.0, social network and mobile Internet, the trend of explosive growth of information is becoming more and more obvious, and the traditional way of information exchange has been greatly impacted. For the current mainstream social media platforms, such as Twitter, Facebook, etc., users can discuss topics of interest and share real-time news anytime and anywhere through the above-mentioned platforms. Due to the huge number of social media users, the simple way to publish information, and the fast speed of information dissemination, social networks contain a wealth of information. However, while social networks bring abundant information, the explosive information also makes it difficult to effecti...

AI Technical Summary

Problems solved by technology

Compared with traditional long text data, social network short text data has the characteristics of large data volume, high noise, low signal-to-noise ratio, irregular expression, and short text length. Therefore, the traditional bag-of-words model that relies on text word frequency information is not applicable. And there will be problems of feature sparsity and dimension disaster

For the above problems, the classification of existing short text data in social networks mainly focuses on the classification research based on semantic features and structural features, but the former needs to rely on large corpus, while the feature selection method of the latter is simple and single, and the selected features can be Scalability and portability are poor, and neither has achieved good results

And the second type of method does not take into account the social network environment where the short text data is located, and does not consider the background characteristics of the text publisher, the statistical syntactic features of the text, and the subsequent impact information of the text on the social network. The beneficial impact of short text data classification

Images