Sensitive word filtering method based on text content

Abstract

The invention discloses a sensitive word filtering method based on text content. The method comprises the steps that a Chinese sensitive word bank is constructed, Chinese words in the Chinese sensitive word bank are expanded to be pinyin blend words, and a pinyin blend sensitive word bank is formed; a transfer function for determining all sensitive words in a finite state automata is established through a sensitive word search tree structure, and sensitive words in the pinyin blend sensitive word bank are made into a sensitive word tree; and the sensitive words are retrieved in a text according to the structure of the sensitive word tree, and the retrieved sensitive words are replaced with designated signs to complete sensitive word filtering. The method is high in recall ratio and easy to implement in practical application.

Description

technical field [0001] The invention relates to the field of filtering sensitive words, in particular to a method for filtering sensitive words based on text content. Background technique [0002] The Internet provides people with a free, convenient and open space, and anyone can speak freely in the virtual world; therefore, a large amount of network information appears to people; One after another, it has caused a serious negative impact on society; sensitive information will be disseminated through various carriers, mainly including pictures, sounds, videos and texts; Carrying out melons cannot achieve good results. The timely discovery, tracking and processing of sensitive information technically plays a pivotal role in reducing the harm of sensitive information on the Internet to society; therefore, in the field of information processing, sensitive information filtering has become a One of the urgent and important technical issues; since English words are separated by s...

AI Technical Summary

Problems solved by technology

[0002] The Internet provides people with a free, convenient and open space, and anyone can speak freely in the virtual world; therefore, a large amount of network information appears to people; One after another, it has caused a serious negative impact on society; sensitive information will be disseminated through various carriers, mainly including pictures, sounds, videos and texts; Carrying out melons cannot achieve good results. The timely discovery, tracking and processing of sensitive information technically plays a pivotal role in reducing the harm of sensitive information on the Internet to society; therefore, in the field of information processing, sensitive information filtering has become a One of the urgent and important technical issues; since English words are separated by spaces, and English only contains 26 letters, there are no obvious separators between Chinese words except for the necessary punctuation marks; and The number of Chinese characters is huge, so many English sensitive information filtering algorithms are not suitable for filtering Chinese sensitive information; if you want to achieve better filtering of sensitive information, you must study a method that can be used in information sources, transmission channels, and receivers. All practical Chinese sensitive information filtering algorithm

[0003] The early text filtering technology is mainly a simple keyword matching and word frequency statistics method, this filtering method is relatively simple; in the single pattern matching algorithm, the more classic ones are: BF algorithm, KMP algorithm, BM algorithm, etc.; in multi-pattern matching Among the algorithms, the more classic ones are: AC algorithm, CW algorithm, WM algorithm, etc. These algorithms can successfully match k...

Images