Accurate filtering method of sensitive words based on artificial intelligence

Abstract

The present invention relates to the field of text processing technology, and in particular to an artificial intelligence-based accurate filtering method for sensitive words, the method comprising: collecting target text sentences to be filtered; using regular expressions and Chinese word segmentation tools to obtain the vocabulary sequence of each target text sentence, and obtaining the suspected sensitive word set of all target text sentences, and then obtaining the writing rationality of each target text sentence; obtaining the sentiment score of each target text sentence according to the BERT model; obtaining the semantic relevance between the suspected sensitive words and the words in the sensitive word dictionary according to the FastText model, and obtaining the semantic desensitization of each sensitive word in the target text sentence, and then obtaining the semantic desensitization of each target text sentence; calculating the potential sensitivity index of each target text sentence, and then filtering the target text sentence. The present invention aims to solve the problem of poor filtering results due to too few features of the target text sentence.

Description

Technical Field

[0001] The present application relates to the technical field of text processing, and in particular to a method for accurately filtering sensitive words based on artificial intelligence. Background Art

[0002] With the rapid development of the Internet, online text content has exploded. However, this is also accompanied by the phenomenon of individual users spreading inappropriate remarks on the Internet, which not only disrupts the order of the Internet, but also may damage the browsing experience of other users. In order to create a clear cyberspace, maintain a healthy online environment, and protect the majority of netizens from the intrusion of harmful information, relevant institutions and organizations must strictly review and filter online texts.

[0003] Thanks to the rapid progress of natural language processing technology, artificial intelligence has matured in text analysis and processing. By using these advanced artificial intelligence technologies, it is possible to effectively screen and purify massive amounts of online text data, thereby preventing the spread of bad information, improving the overall quality of the network environment, and optimizing the user's online experience. In the filtering of sensitive words, regular expression matching, dictionary-based search and other methods can be used to identify and remove potentially harmful content. However, in the face of some users' deliberate evasion of the automatic detection system, such as intentional spelling errors, replacement with synonyms, adding interference characters and other means to change the form of sensitive words, this brings additional challenges to the identification and filtering of sensitive words. Traditional sensitive word recognition and filtering methods have the problem of poor filtering results due to too few features of the target text sentences. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an accurate filtering method for sensitive words based on artificial intelligence to solve the existing problems.

[0005] The sensitive word accurate filtering method based on artificial intelligence of the present invention adopts the following technical scheme:

[0006] An embodiment of the present invention provides a sensitive word accurate filtering method based on artificial intelligence, the method comprising the following steps:

[0007] Collect target text sentences to be filtered;

[0008] Use regular expressions and Chinese word segmentation tools to obtain the vocabulary sequence of each target text sentence; according to the sensitive word dictionary, use regular expressions to obtain the suspected sensitive word set of all target text sentences; obtain the writing rationality of each target text sentence based on the word frequency of the suspected sensitive word in the suspected sensitive word set; obtain the sentiment analysis model based on the BERT model training, and use the sentiment analysis model to calculate the sentiment score of each target text sentence; according to the FastText model, use cosine similarity to obtain the semantic relevance between the suspected sensitive words and the words in the sensitive word dictionary; obtain the semantic desensitization of each sensitive word in the target text sentence based on the semantic relevance, and then obtain the semantic desensitization of each target text sentence; obtain the potential sensitivity index of each target text sentence based on the semantic desensitization, writing rationality and sentiment score of the target text sentence;

[0009] Filter the target text sentences according to the potential sensitivity index.

[0010] Furthermore, the method of using regular expressions and Chinese word segmentation tools to obtain the vocabulary sequence of each target text sentence includes:

[0011] Use regular expressions to extract special characters and punctuation marks in the target text sentence, and delete the special characters and punctuation marks in the target text sentence after extraction;

[0012] A Chinese word segmentation tool is used to perform word segmentation operations on Chinese target text sentences to obtain the vocabulary in each target text sentence, and the sequence composed of the vocabulary in each target text sentence is used as the vocabulary sequence of each target text sentence.

[0013] Furthermore, the method of obtaining a set of suspected sensitive words of all target text sentences using regular expressions according to the sensitive word dictionary includes:

[0014] Use regular expressions to perform fuzzy matching between the words in the word sequence of all target text sentences and the sensitive word dictionary, and regard the words in the word sequence of all target text sentences that are successfully matched as suspected sensitive words; and regard the set of all suspected sensitive words as the suspected sensitive word set.

[0015] Furthermore, the step of obtaining the writing rationality of each target text sentence according to the word frequency of the suspected sensitive word in the suspected sensitive word set includes:

[0016] For each suspected sensitive word in each target text sentence, the frequency of the suspected sensitive word in the suspected sensitive word set is calculated as the word frequency of each suspected sensitive word, the difference between the position of each suspected sensitive word in its target text sentence and the number 1 is calculated, the product of the difference and the word frequency of each suspected sensitive word is calculated, and the average of all products in the target text sentence is calculated as the writing rationality of each target text sentence.

[0017] Furthermore, the sentiment analysis model is obtained according to the BERT model training, and the sentiment analysis model is used to calculate the sentiment score of each target text sentence, including:

[0018] Train the pre-trained BERT model according to the target text sentence to obtain the sentiment analysis model;

[0019] Add a fully connected layer to the outermost layer of the sentiment analysis model to output the sentiment score of the target text sentence;

[0020] The sigmoid activation function is used to limit the value range of the sentiment score to [-1, 1].

[0021] Furthermore, the method of obtaining the semantic relevance between the suspected sensitive words and the words in the sensitive word dictionary using cosine similarity according to the FastText model includes:

[0022] Use the FastText model to obtain the semantic vectors of all suspected sensitive words in the sensitive word dictionary and the vocabulary sequences of all target text sentences;

[0023] The cosine similarity between the semantic vector of each suspected sensitive word in the vocabulary sequence of the target text sentence and each word in the sensitive word dictionary is calculated as the semantic correlation between the suspected sensitive word and the words in the sensitive word dictionary.

[0024] Further, obtaining the semantic desensitization degree of each sensitive word in the target text sentence according to the semantic relevance includes:

[0025] For each suspected sensitive word in each target text sentence, calculate the average of the first k semantic correlations between all words in the sensitive word dictionary and each suspected sensitive word in descending order of semantic correlation, where k is the preset most relevant number;

[0026] The negative value of the calculation result of the logarithmic function with the natural constant as the base and the mean as a true number is calculated as the semantic desensitization degree of each suspected sensitive word in each target text sentence.

[0027] Further, the obtaining of the semantic desensitization degree of each target text sentence includes:

[0028] The mean of the semantic desensitization degree of each suspected sensitive word in each target text sentence is taken as the semantic desensitization degree of each target text sentence.

[0029] Furthermore, the method of obtaining the potential sensitivity index of each target text sentence according to the semantic desensitization degree, writing rationality and sentiment score of the target text sentence includes:

[0030] For each target text sentence, the product of the semantic desensitization degree and the writing rationality of the target text sentence is calculated as the first product;

[0031] Calculate the difference between the absolute value of the sentiment score of the target text sentence and the number 1, calculate the calculation result of an exponential function with a natural constant as the base and the difference as the exponent, and calculate the product of the number of suspected sensitive words in the target text sentence and the calculation result as the second product;

[0032] The ratio of the second product to the first product is taken as the potential sensitivity index of each target text sentence.

[0033] Further, filtering the target text sentences according to the potential sensitivity index includes:

[0034] The target text sentence and label are used as input of the LSTM model, and the output of the model is the inference result;

[0035] Among them, the labels include "sensitive meaning" and "non-sensitive meaning", and the inference results are "is a sensitive target text sentence" or "is not a sensitive target text sentence", and the suspected sensitive words in the target text sentences with the result of "is a sensitive target text sentence" are filtered.

[0036] The present invention has at least the following beneficial effects:

[0037] The present invention analyzes and extracts features from the target text sentences, and then trains a more accurate classification model for sensitive target text sentences, and finally achieves the purpose of filtering sensitive words in the data. Specifically, firstly, the text structure anomaly is eliminated by preprocessing, and then a set of suspected sensitive words is constructed using a sensitive word dictionary and the writing rationality of the target text sentences is calculated, and then the potential sensitivity index of the target text sentences is determined according to the writing rationality and semantic relevance of the target text sentences. Finally, the potential sensitivity index can be used as an additional feature of the target text sentence for training a classification model for sensitive target text sentences to improve the accuracy and robustness of the model. Such a model can better distinguish target text sentences that contain sensitive words but do not actually express sensitive meanings, and filter sensitive words according to the classification results, solving the problem of poor filtering results caused by considering too few features of the target text sentences. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flowchart of the sensitive word accurate filtering method based on artificial intelligence provided by the present invention;

[0040] Figure 2 Get the flowchart for the sensitive target text sentence classification model. DETAILED DESCRIPTION

[0041] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of the sensitive word accurate filtering method based on artificial intelligence proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0042] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0043] The specific scheme of the sensitive word accurate filtering method based on artificial intelligence provided by the present invention is described in detail below with reference to the accompanying drawings.

[0044] An embodiment of the present invention provides an accurate filtering method for sensitive words based on artificial intelligence. Specifically, the following accurate filtering method for sensitive words based on artificial intelligence is provided. Figure 1 , the method comprises the following steps:

[0045] Step S001, obtaining a target text sentence.

[0046] On social media platforms, users generate a huge amount of content every day. These contents usually contain a lot of sensitive information, so sensitive words need to be detected and filtered.

[0047] This embodiment uses sentences in the Chinese offensive language detection public dataset COLD as target text sentences and analyzes the target text sentences.

[0048] Among them, the Chinese Offensive Language Detection Public Dataset COLD is a dataset used to detect offensive language in Chinese texts. The dataset contains more than 100,000 annotated data from different platforms and fields.

[0049] At this point, the target text sentence is obtained.

[0050] Step S002, to ensure the compliance of the text content, first eliminate text structure anomalies through preprocessing, then use the sensitive word dictionary to construct a set of suspected sensitive words, calculate the writing rationality of the target text sentence, and then determine the potential sensitivity index of the target text sentence based on the writing rationality and semantic relevance of the target text sentence.

[0051] In the process of identifying and filtering sensitive words, some sensitive words may be changed to a certain extent by adding punctuation marks, spaces or special characters that do not fit the context in order to circumvent the sensitive word detection system.

[0052] Therefore, this embodiment first cleans the target text sentences in the Chinese offensive language detection public data set, uses regular expressions to extract special characters and punctuation marks in the target text sentences, and deletes the special characters and punctuation marks of the target text sentences after extraction, such as "@", "#", "￥", etc. Further, the Chinese word segmentation tool jieba of Python is used to perform word segmentation operations on the Chinese target text sentences. After the word segmentation is completed, the vocabulary in each target text sentence is converted into a series of vocabulary sequences separated by spaces. Among them, the process of extracting special characters and punctuation marks in the target text sentences by regular expressions and the jieba Chinese word segmentation tool are well-known technologies and will not be elaborated here.

[0053] Furthermore, it is possible to encounter target text sentences with abnormal structures, which may lead to misjudgment. Users may circumvent the monitoring of the sensitive word detection system by changing the form of sensitive words. For example, by replacing sensitive words with homophones or synonyms, changing the original form of sensitive words to avoid detection by the automatic review system. Some words that are usually considered sensitive may not violate the rules in specific scenarios. The meaning and usage context of these words need to be carefully analyzed in combination with the specific context to ensure that improper review will not be caused by misunderstanding the true intention of their expression.

[0054] First, obtain a sensitive word dictionary from public resources. This embodiment uses the fastscan library as the sensitive word dictionary W of this invention, where fastscan is an open source library on the github website. It contains sensitive words in various fields.

[0055] Furthermore, considering that users may deliberately modify the composition of sensitive words, thereby changing the form of sensitive words to evade the sensitive word detection system, regular expressions are used here to fuzzy match the words in the vocabulary sequence of all target text sentences with the words in the sensitive word dictionary W to identify the deformed suspected sensitive words in the vocabulary sequence of all target text sentences. Among them, regular expressions for fuzzy matching can construct regular expressions according to the rules of sensitive words. For example, replacing "sensitive words" with "sensitive {0,3} words" means that there can be 0 to 3 arbitrary characters between "sensitive" and "words". The use of regular expressions is a well-known technology and will not be repeated here.

[0056] Finally, the set of suspected sensitive words in the vocabulary sequences of all target text sentences obtained by regular expression matching is taken as the suspected sensitive word set A.

[0057] First, all target text sentences in the Chinese offensive language detection public dataset are traversed, and the frequency of each suspected sensitive word in the suspected sensitive word set A in each target text sentence is calculated as the word frequency of each suspected sensitive word. According to the word frequency of the suspected sensitive word and the number of suspected sensitive words, the writing rationality β of each target text sentence is calculated, where the position of the suspected sensitive word in the target text sentence is recorded as PosA, where the position of the suspected sensitive word in the target text sentence is its serial number in the vocabulary sequence of the target text sentence.

[0058]

[0059] Among them, β i Indicates the writing rationality of the i-th target text sentence; N i represents the number of suspected sensitive words in the i-th target text sentence; FA i,j PosA represents the frequency of the jth suspected sensitive word in the i-th target text sentence; i,j represents the position of the jth suspected sensitive word in the i-th target text sentence; N i Represents the number of suspected sensitive words in the i-th target text sentence.

[0060] Formula logic: The more suspected sensitive words there are in the target text sentence, the more likely it is that the target text sentence has inappropriate descriptions, that is, the lower the writing rationality, and vice versa, the higher the writing rationality; FA i,j represents the frequency of the jth suspected sensitive word in the i-th target text sentence. When the frequency of the suspected sensitive word is lower, it indicates that the suspected sensitive word is more likely to be a disguised word that the user deliberately changes its form, and the writing rationality of the target text sentence is lower. On the contrary, it indicates that the suspected sensitive word is a normal word that is frequently used, and the writing rationality of the target text sentence is higher. When PosA i,j The smaller the -1 is, the more concentrated the suspected sensitive words are in the target text sentences, the more likely they are sensitive words, and the lower the writing rationality of the target text sentences. Conversely, the more discrete the distribution of suspected sensitive words is, the less likely they are sensitive words, and the higher the writing rationality of the target text sentences.

[0061] When determining whether a target text sentence is a sensitive target text sentence, the same word may have different meanings in different contexts. For example, a sensitive word may simply describe a concept when mentioned in an academic field, but may have a different meaning when it comes to sensitive topics. Therefore, this embodiment analyzes the meaning of the target text sentence from multiple directions.

[0062] First, the pre-trained BERT model is trained based on the target text sentences in the Chinese offensive language detection public dataset to obtain fine-tuned BERT. Among them, fine-tuning BERT means further training the pre-trained BERT model using data related to specific fields or tasks to adapt to the needs of specific tasks.

[0063] Specifically, a pre-trained BERT model is selected. The pre-trained model selected in this embodiment is BERT-Base. The implementer can select other pre-trained models according to actual conditions. Each target text sentence in the Chinese offensive language detection public data set has a corresponding label, and the labels include "sensitive meaning" and "non-sensitive meaning". The data in the Chinese offensive language detection public data set and its corresponding labels are used to load parameters from the pre-trained BERT model. The parameters include the rich language representation learned from the data in the Chinese offensive language detection public data set and its corresponding labels, which will not be described in detail here. The subsequent pre-training process is a technology well known to those skilled in the art and will not be described in detail here.

[0064] After completing the training of the pre-trained BERT model, the fine-tuned BERT is used as a sentiment analysis model, and a fully connected layer is added to the outermost layer of the sentiment analysis model to output the sentiment score of the target text sentence. The sigmoid activation function is used to limit the value range of the sentiment score to [-1,1]. Among them, when the sentiment score is closer to 0, it means that the target text sentence expresses less emotion; when the sentiment score deviates from 0, it means that the target text sentence expresses richer emotions, that is, it is more likely to contain sensitive meanings.

[0065] Specifically, the input of the sentiment analysis model is each target text sentence, and the output is the sentiment score corresponding to each target text sentence. The sentiment score of the i-th target text sentence is S i Among them, the BERT model and the sigmoid activation function are well-known technologies in this field and will not be described in detail here.

[0066] Furthermore, this embodiment uses the FastText model to obtain the semantic vectors of each suspected sensitive word in the vocabulary sequence of the sensitive word dictionary W and the target text sentence, wherein the FastText model is a well-known model and will not be described in detail here; the semantic vector refers to the technology of converting text into a mathematical vector representation, which captures the meaning and semantic information of words, phrases or entire sentences by mapping them to points in the vector space, which will not be described in detail here.

[0067] Furthermore, the semantic vector of each suspected sensitive word in the vocabulary sequence of the target text sentence and the cosine similarity between each word in the sensitive word dictionary W are calculated as the semantic relevance; wherein, the calculation method of the cosine similarity is a well-known technology and will not be described in detail here.

[0068] Furthermore, the semantic desensitization degree ρ of the suspected sensitive words in the target text sentence is constructed, and the specific calculation is as follows:

[0069]

[0070]

[0071] Among them, ρ i,j represents the semantic desensitization degree of the jth suspected sensitive word in the vocabulary sequence of the i-th target text sentence; ρ i represents the semantic desensitization degree of the i-th target text sentence; k represents the k words in the sensitive word dictionary W that are most relevant to the semantic relevance of the j-th suspected sensitive word in the vocabulary sequence of the i-th target text sentence, and the empirical value of k is 5; sim(A i,j ,W) represents the semantic relevance between the jth suspected sensitive word in the i-th target text sentence and all the words in the sensitive word dictionary W; max l () indicates the lth most relevant semantic relevance; N i Represents the number of suspected sensitive words in the i-th target text sentence.

[0072] Formula logic: The greater the semantic similarity between the suspected sensitive words in the i-th target text sentence and the sensitive dictionary, the more likely the target text sentence is to be a sensitive target text sentence, and the smaller the semantic desensitization degree is. Conversely, the more likely the target text sentence is not to be a sensitive target text sentence, and the greater the semantic desensitization degree is.

[0073] The potential sensitivity index of each target text sentence is calculated based on the semantic desensitization and sentiment score of the target text sentence, as follows:

[0074]

[0075] Among them, α i represents the potential sensitivity index of the i-th target text sentence; N i represents the number of suspected sensitive words in the i-th target text sentence; S i represents the sentiment score of the i-th target text sentence, and its value is [-1,1]. When the sentiment score is less than 0, it indicates that the sentence expresses a negative sentiment. When the sentiment score is greater than or equal to 0, it indicates that the sentence expresses a positive sentiment. The larger the absolute value, the stronger the sentiment expression. i represents the semantic desensitization degree of the i-th target text sentence; β i Indicates the writing rationality of the i-th target text sentence.

[0076] Formula logic: When the i-th target text sentence has more suspected sensitive words, it indicates that the target text sentence is more likely to contain sensitive topics, that is, the potential sensitivity index of the target text sentence is larger, otherwise, its potential sensitivity index is smaller; when the semantic desensitization degree is larger, it indicates that the suspected sensitive words in the target text sentence are less likely to express sensitive meanings in the current scenario, that is, the potential sensitivity index of the target text sentence should be smaller, otherwise, it indicates that the suspected sensitive words are more likely to express sensitive meanings in the current scenario, that is, the potential sensitivity index of the target text sentence should be larger; when S i The closer it is to zero, the smaller the emotion expressed by the target text sentence, that is, the less likely it is to contain sensitive meanings, and the smaller its potential sensitivity index should be. On the contrary, when S i The further away from zero, the richer the emotion expressed by the target text sentence, that is, the more likely it is to contain sensitive meanings, and the larger its potential sensitivity index should be; when the writing rationality of the i-th target text sentence is greater, it indicates that the target text sentence is more likely to be a normal description of the target text sentence, and its potential sensitivity index should be smaller, otherwise, its potential sensitivity index should be larger.

[0077] Step S003: train a sensitive target text sentence classification model according to the potential sensitivity index and filter suspected sensitive words.

[0078] Use the LSTM model to add a classification layer to the outermost layer to build a sensitive target text sentence classification model to classify the target text sentences. The sensitive target text sentence classification model is obtained as follows: Figure 2 As shown in the figure, the LSTM model is a model well known to those skilled in the art and will not be described in detail here.

[0079] The sensitive target text sentence classification model uses Adam as the optimizer, and the loss function uses the cross entropy loss. In the training phase, the input is the target text sentence containing labels, the labels include "sensitive meaning" and "non-sensitive meaning", and the potential sensitivity index α corresponding to the target text sentence, and the output is the reasoning result. Among them, the specific training process of the LSTM model is a prior art, and this embodiment will not be repeated here.

[0080] Among them, the input layer is responsible for receiving input data, then representing it as a corresponding semantic vector and passing the result to the next layer; the LSTM layer is composed of a long short-term memory network, which is used to capture long-term dependencies in the input sequence; the fully connected layer is responsible for receiving the output of the previous layer, and then further characterizing the characteristics of the input sequence; α is the potential sensitivity index of the current input target text sentence sequence, indicating the possibility that the target text sentence is a sensitive target text sentence, and its potential sensitivity index is used as a weight to multiply each neuron in the last layer of the fully connected layer, so as to achieve the purpose of participating in model training; the Softmax layer is responsible for converting the output of the fully connected layer 2 through the softmax function to obtain the probability distribution of each category; the output layer uses appropriate strategies based on the output of the softmax layer, such as selecting the category with the highest probability, to generate the final classification prediction result. Among them, the softmax function is a well-known technology and will not be described here.

[0081] For the trained model, in the inference stage, the input is a target text sentence to be detected, and the output is the result of its inference, which is "a sensitive target text sentence" or "not a sensitive target text sentence". Finally, according to the classification result, the suspected sensitive words in the target text sentence with the result of "a sensitive target text sentence" are filtered to complete the accurate filtering of sensitive words.

[0082] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0083] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0084] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Modifications to the technical solutions recorded in the aforementioned embodiments, or equivalent replacement of some of the technical features therein, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. The sensitive word accurate filtering method based on artificial intelligence is characterized by: The method comprises the following steps: Collect target text sentences to be filtered; Use regular expressions and Chinese word segmentation tools to obtain the vocabulary sequence of each target text sentence; according to the sensitive word dictionary, use regular expressions to obtain the suspected sensitive word set of all target text sentences; obtain the writing rationality of each target text sentence based on the word frequency of the suspected sensitive word in the suspected sensitive word set; obtain the sentiment analysis model based on the BERT model training, and use the sentiment analysis model to calculate the sentiment score of each target text sentence; according to the FastText model, use cosine similarity to obtain the semantic relevance between the suspected sensitive words and the words in the sensitive word dictionary; obtain the semantic desensitization of each sensitive word in the target text sentence based on the semantic relevance, and then obtain the semantic desensitization of each target text sentence; obtain the potential sensitivity index of each target text sentence based on the semantic desensitization, writing rationality and sentiment score of the target text sentence; Filter the target text sentences according to the potential sensitivity index; The step of obtaining the writing rationality of each target text sentence according to the word frequency of the suspected sensitive word in the suspected sensitive word set includes: For each suspected sensitive word in each target text sentence, the frequency of the suspected sensitive word in the suspected sensitive word set is calculated as the word frequency of each suspected sensitive word, the difference between the position of each suspected sensitive word in its target text sentence and the number 1 is calculated, the product of the difference and the word frequency of each suspected sensitive word is calculated, and the average of all products in the target text sentence is calculated as the writing rationality of each target text sentence; The method of obtaining the potential sensitivity index of each target text sentence according to the semantic desensitization degree, writing rationality and sentiment score of the target text sentence includes: For each target text sentence, the product of the semantic desensitization degree and the writing rationality of the target text sentence is calculated as the first product; Calculate the difference between the absolute value of the sentiment score of the target text sentence and the number 1, calculate the calculation result of an exponential function with a natural constant as the base and the difference as the exponent, and calculate the product of the number of suspected sensitive words in the target text sentence and the calculation result as the second product; The ratio of the second product to the first product is taken as the potential sensitivity index of each target text sentence.

2. The method for accurate filtering of sensitive words based on artificial intelligence as claimed in claim 1, characterized in that: The method of using regular expressions and Chinese word segmentation tools to obtain the vocabulary sequence of each target text sentence includes: Use regular expressions to extract special characters and punctuation marks in the target text sentence, and delete the special characters and punctuation marks in the target text sentence after extraction; A Chinese word segmentation tool is used to perform word segmentation operations on Chinese target text sentences to obtain the vocabulary in each target text sentence, and the sequence composed of the vocabulary in each target text sentence is used as the vocabulary sequence of each target text sentence.

3. The method for accurate filtering of sensitive words based on artificial intelligence as claimed in claim 1, characterized in that: According to the sensitive word dictionary, a regular expression is used to obtain a set of suspected sensitive words in all target text sentences, including: Use regular expressions to perform fuzzy matching between the words in the word sequence of all target text sentences and the sensitive word dictionary, and regard all successfully matched words in the word sequence of all target text sentences as suspected sensitive words; and regard the set of all suspected sensitive words as the suspected sensitive word set.

4. The method for accurate filtering of sensitive words based on artificial intelligence as claimed in claim 1, characterized in that: The sentiment analysis model is obtained by training the BERT model, and the sentiment analysis model is used to calculate the sentiment score of each target text sentence, including: Train the pre-trained BERT model according to the target text sentence to obtain the sentiment analysis model; Add a fully connected layer to the outermost layer of the sentiment analysis model to output the sentiment score of the target text sentence; The sigmoid activation function is used to limit the value range of the sentiment score to [-1, 1].

5. The method for accurate filtering of sensitive words based on artificial intelligence as claimed in claim 1, characterized in that: According to the FastText model, the cosine similarity is used to obtain the semantic relevance between the suspected sensitive words and the words in the sensitive word dictionary, including: Use the FastText model to obtain the semantic vectors of all suspected sensitive words in the sensitive word dictionary and the vocabulary sequences of all target text sentences; The cosine similarity between the semantic vector of each suspected sensitive word in the vocabulary sequence of the target text sentence and each word in the sensitive word dictionary is calculated as the semantic correlation between the suspected sensitive word and the words in the sensitive word dictionary.

6. The method for accurate filtering of sensitive words based on artificial intelligence as claimed in claim 1, characterized in that: The step of obtaining the semantic desensitization degree of each sensitive word in the target text sentence according to the semantic relevance includes: For each suspected sensitive word in each target text sentence, calculate the average of the first k semantic correlations between all words in the sensitive word dictionary and each suspected sensitive word in descending order of semantic correlation, where k is the preset most relevant number; The negative value of the calculation result of the logarithmic function with the natural constant as the base and the mean as a true number is calculated as the semantic desensitization degree of each suspected sensitive word in each target text sentence.

7. The method for accurate filtering of sensitive words based on artificial intelligence as claimed in claim 1, characterized in that: The obtaining of the semantic desensitization degree of each target text sentence includes: The mean of the semantic desensitization degree of each suspected sensitive word in each target text sentence is taken as the semantic desensitization degree of each target text sentence.

8. The method for accurate filtering of sensitive words based on artificial intelligence as claimed in claim 1, characterized in that: The filtering of the target text sentences according to the potential sensitivity index includes: The target text sentence and label are used as input of the LSTM model, and the output of the model is the inference result; Among them, the labels include "sensitive meaning" and "non-sensitive meaning", the inference results are "is a sensitive target text sentence" or "is not a sensitive target text sentence", and the suspected sensitive words in the target text sentence with the result of "is a sensitive target text sentence" are filtered.

Images