Mass short message information filtering method based on semantic extension

Abstract

The invention provides a mass short message information filtering method based on semantic extension and can solve the short message feather sparsity problem. The method comprises the following steps that 1, an initial training sample set is built, and expansion is carried out for each sample of the training sample set on the basis of the context information; 2, the expanded training sample set is subjected to text preprocessing; 3, a theme feather dictionary is built on the basis of the training sample set subjected to the preprocessing; 4, each text of the training sample set is subjected to text expression in a hidden theme space; 5, an SVM (support vector machine) filter is built; 6, the text to be filtered is subjected to expansion and text preprocessing to be converted into a feather word set on the basis of the context information, is then subjected to text expression in the hidden theme space, and is filtered by a filter; and 7, novel samples are regularly collected, the word item probability distribution of the theme is updated in the existing hidden theme space, the novel samples are subjected to text expression, and the SVM filter is rebuilt.

Description

technical field [0001] The invention belongs to the technical field of information filtering, in particular to a method for filtering massive short text information based on semantic expansion. Background technique [0002] In recent years, new media represented by the Internet and mobile phones are playing an increasingly important role in people's daily life, study and work. People can pay attention to social hotspots and participate in social public affairs through Weibo, SMS, news comments, etc. The powerful communication function and influence of public opinion possessed by new media are extensively and profoundly affecting all aspects of human society. However, with the positive development of new media, there are also some negative phenomena that cannot be ignored. Some people use new media to wantonly spread reactionary speeches, vulgar false information, etc., and some companies or individuals also take the opportunity to distribute a large number of advertisements...

AI Technical Summary

Problems solved by technology

[0003] However, most of the existing information filtering technologies determine the feature space based on word frequency. This method is more suitable for long text information, while information from Weibo, SMS, news comments, etc. all exist in the form of short text. , the shortness of the text content makes there are few effective features that can be used, and there are few common features between different texts. The feature sparsity of short texts will directly affect the effectiveness of information filtering.

Secondly, the size of the training sample data set used by the existing information filtering technology is relatively small, and the size of the training sample set required for short text information filtering is much larger to ensure that the distribution of the actual data is consistent. , and the running speed of existing learning algorithms on large sample data sets will definitely become a huge bottleneck

Another very important issue is how the filter responds to changing data. Existing information filtering technologies either ignore data changes or use incremental learning strategies to solve them, but these incremental learning strategies are mostly based on The same feature space is carried out, and the change of the actual data is more that the feature space has changed. For the data with this feature space changed, the existing information filtering technology is almost powerless.

Images