Short text auditing method and device fusing variant word recognition

Abstract

The invention belongs to the field of short text auditing, particularly relates to a short text auditing method and device fusing variant word recognition, and aims to solve the problem of how to fusea variant word recognition technology into a harmful text auditing task and realize automatic updating of a model. The method comprises the steps of constructing a configuration word bank, obtainingto-be-audited text data based on a social media platform, screening the to-be-audited text data to obtain suspicious text data, removing meaningless information, and calculating a text feature vectorand a statistical feature vector; and performing feature fusion on the text feature vectors and the statistical feature vectors, obtaining harmful texts through a trained harmful text classification model based on a support vector machine, obtaining sensitive words of the harmful texts by utilizing a preset keyword extraction algorithm, and writing the sensitive words into a configuration word bank. According to the method, the variant word recognition technology is fused into text feature and statistical feature calculation to carry out harmful text auditing tasks, automatic model updating isachieved, and the text auditing accuracy and updating speed are improved.

Description

technical field [0001] The invention belongs to the field of text analysis, and in particular relates to a short text review method and device for fusion variant word recognition. Background technique [0002] With the gradual maturity of various social media platforms such as Twitter and Weibo, the threshold for information dissemination is gradually lowered, and users can easily disseminate information on the Internet. While the scale of user contributions continues to grow, content chaos is also becoming increasingly prominent. Some lawbreakers use social media to spread politically sensitive, maliciously promoted, pornographic and violent content. These harmful contents not only affect the user experience, but also bring great legal risks to the platform and have a very bad impact on the network environment. Therefore, how to identify and filter harmful content from massive amounts of information has become an important issue. [0003] The traditional content review m...

AI Technical Summary

Problems solved by technology

The disadvantages of manual review are also obvious: (1) The iteration of harmful information content is fast, and the update of sensitive thesaurus is delayed

(2) The way of matching sensitive words may "accidentally injure" some harmless texts, so a second review is required manually

(3) The amount of harmful text information is large, and the cost of manual review is high

[0004] Later, a text review method based on machine learning appeared in the industry, which reduced the cost of manual review to a certain extent. This type of method has the following disadvantages: (1) The accuracy of the harmful short text classification method based on traditional machine learning is not high, which is Due to the short length and low content of social media information, traditional machine learning methods are very easy to accidentally damage some harmless short texts containing sensitive words

(2) Harmful information has the characteristics of irregular expression. Information publishers will replace sensitive words with variants (such as homonyms) of sensitive words. Text review methods based on machine learning do not have the ability to identify variants

(3) The topic and content of harmful information are updated quickly, and the model needs to be updated frequently to ensure the recall rate of the model

Images