Sensitive word recognition method based on big data

Abstract

The invention discloses a sensitive word recognition method based on big data. The method comprises the steps of 1, collecting text data by utilizing crawler software, performing sensitive marking on the text data to obtain a sensitive text D1 and a normal text D2, performing sensitive word classification and grade marking on sensitive words, and storing the sensitive words into a sensitive word list S; 2, carrying out new word discovery through an N-gram model, and amplifying the sensitive word lists S; and 3, carrying out deformation processing on each sensitive word in each sensitive word list S to obtain deformed sensitive words. 4, filtering the sensitive words in the sensitive word list S based on a Trie tree and a BERT model; according to the method, the accuracy and efficiency of auditing and identifying the sensitive words are improved.

Description

technical field [0001] The invention relates to the field of big data research, in particular to a method of natural language processing, in particular to a method for identifying sensitive words based on big data. Background technique [0002] With the continuous development of the Internet, people can see a large amount of text information through various platforms of the Internet. Some of the information contains sensitive information, such as terrorist pornography, etc. If it is not identified and controlled, it will hinder social stability and damage social and public interests. It is an important task to control the quality of text information, identify and process these sensitive information in a timely manner, ensure that the published content does not contain sensitive information, and create a healthy network environment. [0003] In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art:...

AI Technical Summary

Problems solved by technology

[0003] In the process of realizing the present invention, the inventors found that there are at least the following problems in the prior art: the traditional technology mainly adopts the method of rule matching for identification, that is, constructing a vocabulary of sensitive words, which mainly comes from manual operation and collection. The cost is high and the efficiency is low. By traversing the vocabulary of sensitive words and matching with each text information, if any sensitive words are found, they need to be submitted to the reviewers for manual review

This method has disadvantages: first, as the number of sensitive words increases, there will be more and more variants of sensitive words, and the vocabulary of sensitive words will also become larger and larger, and the search speed will slow down due to circular matching; second, the The method still requires reviewers to make manual judgments, and the manual workload is heavy; third, this method can only detect whether sensitive words appear or not, ignoring the context in which sensitive words appear, which is likely to cause false positives

Images