Sensitive word detection method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the technical field of natural language processing, and relates to a sensitive word detection method and device, computer equipment and a storage medium, and the method comprises the following steps: receiving a sensitive word detection request sent by a user terminal; calling a Chinese character Unicode code white list, and performing interference character clearing operation on the input text data according to the Chinese character Unicode code white list to obtain pure text data; performing character merging operation on the pure text data to obtain to-be-matched text data; calling a sensitive word Trie tree, and performing sensitive word similarity matching operation on the sensitive word Trie tree and characters of the to-be-matched text data in a similarity matching chain by taking a root node character of the sensitive word Trie tree as a starting point to obtain matching result data; performing comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result; and outputting the comprehensive scoring result to the user terminal. According to the method, the sensitive words in the text input by the user are automatically and efficiently recognized and filtered, so that the safety and compliance of the system are protected.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a sensitive word detection method, apparatus, computer equipment and storage medium. Background Art

[0002] The technology for precise keyword matching is already very mature, for example: Trie tree algorithm, AC automaton algorithm, dual-array trie tree algorithm, dual-array AC automaton algorithm, etc.

[0003] However, the applicant found that although the algorithm of accurate keyword matching is very applicable in the field of accurate keyword search, it is not practical in the scenario where fuzzy matching of keywords is required. For example, in the field of SMS sending, bad information is often sent to users' mobile phones. These bad information often contain variant sensitive words to bypass the sensitive content check before sending the information. Figure 3 The schematic diagram of sensitive content shown in the figure replaces some characters of the original Chinese sensitive word with other Chinese characters with similar pronunciation, similar shape, traditional Chinese characters, etc., and mixes in some special characters, making the variants of a sensitive word countless.

[0004] This shows that traditional precise keyword matching technology cannot be applied to scenarios with fuzzy matching keywords. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a sensitive word detection method, device, computer equipment and storage medium to solve the problem that traditional keyword precise matching technology cannot be applied to the scenario of fuzzy matching keywords.

[0006] In order to solve the above technical problems, the present application embodiment provides a sensitive word detection method, which adopts the following technical solution:

[0007] Receiving a sensitive word detection request sent by a user terminal, wherein the sensitive word detection request includes input text data to be detected;

[0008] Calling a Chinese character Unicode encoding whitelist, and performing an interference character removal operation on the input text data according to the Chinese character Unicode encoding whitelist to obtain pure text data;

[0009] Performing a character merging operation on the clean text data to obtain text data to be matched;

[0010] Calling the sensitive word Trie tree, and taking the root node character of the sensitive word Trie tree as the starting point, respectively performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain, to obtain matching result data;

[0011] Performing a comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result;

[0012] Output the comprehensive scoring result to the user terminal.

[0013] Furthermore, the step of performing an interference character removal operation on the input text data according to the Chinese character Unicode encoding whitelist to obtain pure text data specifically includes the following steps:

[0014] Obtaining a target Unicode block corresponding to the input text data in the Chinese character Unicode encoding whitelist;

[0015] Performing an interference character matching operation on the input text data according to the target Unicode block to obtain an interference character matching result;

[0016] The interference characters in the input text data are removed according to the interference character matching result to obtain the pure text data.

[0017] Furthermore, the step of performing a character merging operation on the clean text data to obtain the text data to be matched specifically includes the following steps:

[0018] Calling a ligature library, and constructing a hash table according to the ligature library;

[0019] Selecting two consecutive characters in the clean text data in sequence, and concatenating the consecutive characters into a character string;

[0020] Performing a hash matching operation in the hash table using the character string as a key to obtain a hash matching result;

[0021] According to the hash matching result, the corresponding characters in the clean text data are replaced with the corresponding values ​​in the hash table to obtain the text data to be matched.

[0022] Furthermore, before the step of taking the root node character of the sensitive word Trie tree as the starting point and performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data, the following steps are also included:

[0023] Constructing an original similarity matching chain, and performing an initialization operation on the original similarity matching chain to obtain an initialized similarity matching chain;

[0024] Constructing an original character precise matching model, and adding the original character precise matching model to the initialized similarity matching chain, wherein the type of the original character precise matching model is 1 and the binary value is 00000001;

[0025] Read the data in the correction character library, build a correction character precise matching model according to the data in the correction character library, and add the correction character precise matching model to the initialization similarity matching chain, wherein the type of the correction character precise matching model is 4, and the binary number is 00000100;

[0026] Read the data in the Zhengma character library, construct a Zhengma character similarity matching model according to the data in the Zhengma character library, and add the Zhengma character similarity matching model to the initialized similarity matching chain, wherein the type of the Zhengma character similarity matching model is 8, and the binary is 00001000;

[0027] Read the data in the Cangjie character library, construct a Cangjie character similarity matching model according to the data in the Cangjie character library, and add the Cangjie character similarity matching model to the initialized similarity matching chain, wherein the type of the Cangjie character similarity matching model is 16, which is 00010000 in binary;

[0028] Read data in the Wubi character library, construct a Wubi character similarity matching model based on the data in the Wubi character library, and add the Wubi character similarity matching model to the initialized similarity matching chain to obtain the similarity matching chain, wherein the type of the Wubi character similarity matching model is 32, and the binary number is 00100000.

[0029] Furthermore, before the step of taking the root node character of the sensitive word Trie tree as the starting point and performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data, the following steps are also included:

[0030] Initialize a root node, wherein the root node does not contain specific characters;

[0031] Obtain a sensitive word string to be inserted, and insert the sensitive word string to be inserted character by character starting from the root node;

[0032] When the characters of the sensitive word character strings to be inserted are all inserted, the sensitive word Trie tree is obtained.

[0033] Furthermore, the step of obtaining the sensitive word string to be inserted, and inserting the sensitive word string to be inserted character by character starting from the root node, specifically includes the following steps:

[0034] Determine whether there is a repeated child node corresponding to the character to be inserted in the current node;

[0035] If the repeated child node does not exist, create a new child node, link the new child node to the current node, record the character to be inserted, and set the matching model of the character to be inserted to the matching model corresponding to the sensitive word;

[0036] If the repeated sub-node exists, move to the repeated sub-node, perform an "OR" operation with the matching model of the character to be inserted and the matching model corresponding to the sensitive word, and set the operation result as the matching model of the character to be inserted.

[0037] Furthermore, the step of taking the root node character of the sensitive word Trie tree as the starting point and performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data specifically includes the following steps:

[0038] Set the current node as the root node;

[0039] Traverse each character of the text data to be matched from the head, and match the nodes of the sensitive word Trie tree from two dimensions of depth and width;

[0040] If each character of the to-be-matched text data is traversed successfully and the end node of the matching sensitive word is reached, it means that the matching sensitive word is successfully matched with the to-be-matched text data, and the target sensitive word is obtained;

[0041] If no matching characters or nodes can be found during the traversal process, or if the end node of any sensitive word is not reached after traversing the text data to be matched, it means that the match fails and there is no target sensitive word;

[0042] The matched target sensitive words are recorded to obtain the matching result data.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a sensitive word detection device, which adopts the following technical solution:

[0044] A request acquisition module, configured to receive a sensitive word detection request sent by a user terminal, wherein the sensitive word detection request includes input text data to be detected;

[0045] An interference character removal module is used to call a Chinese character Unicode encoding white list, and perform an interference character removal operation on the input text data according to the Chinese character Unicode encoding white list to obtain pure text data;

[0046] A character merging module, used for performing a character merging operation on the clean text data to obtain text data to be matched;

[0047] A sensitive word similarity matching module is used to call a sensitive word Trie tree, and use the root node character of the sensitive word Trie tree as a starting point to perform sensitive word similarity matching operations with the characters of the text data to be matched in a similarity matching chain to obtain matching result data;

[0048] A comprehensive scoring module is used to perform a comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result;

[0049] A result output module is used to output the comprehensive scoring result to the user terminal.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0051] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the sensitive word detection method as described above when executing the computer-readable instructions.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0053] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the sensitive word detection method described above are implemented.

[0054] The present application provides a sensitive word detection method, comprising: receiving a sensitive word detection request sent by a user terminal, wherein the sensitive word detection request includes input text data to be detected; calling a Chinese character Unicode encoding whitelist, and performing an interference character removal operation on the input text data according to the Chinese character Unicode encoding whitelist to obtain pure text data; performing a character merging operation on the pure text data to obtain text data to be matched; calling a sensitive word Trie tree, and using the root node character of the sensitive word Trie tree as a starting point to perform a sensitive word similarity matching operation with the characters of the text data to be matched in a similarity matching chain to obtain matching result data; performing a comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result; and outputting the comprehensive scoring result to the user terminal. Compared with the prior art, the present application automatically and efficiently identifies and filters sensitive words in user input text, thereby protecting the security and compliance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0057] Figure 2 is a flowchart of the implementation of the sensitive word detection method provided in the embodiment of the present application;

[0058] Figure 3 It is a schematic diagram of sensitive content provided by the background technology of this application;

[0059] Figure 4 is a schematic diagram of a Unicode encoding whitelist provided in an embodiment of the present application;

[0060] Figure 5 is a schematic diagram of a combined character library provided in an embodiment of the present application;

[0061] Figure 6 is a schematic diagram of a process for constructing a similarity matching chain provided in an embodiment of the present application;

[0062] Figure 7 is a schematic diagram of a sensitive word library provided in an embodiment of the present application;

[0063] Figure 8 It is a schematic diagram of a process for similarity matching of sensitive words in a text provided by an embodiment of the present application;

[0064] Fig. 9 is a schematic diagram of the structure of a sensitive word detection device provided in an embodiment of the present application;

[0065] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0069] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0070] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0071] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0072] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0073] It should be noted that the sensitive word detection method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the sensitive word detection device is generally set in the server / terminal device.

[0074] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0075] Continue to refer Figure 2 , shows a flow chart of an embodiment of a sensitive word detection method according to the present application. The sensitive word detection method comprises: step S201, step S202, step S203, step S204, step S205 and step S206.

[0076] In step S201, a sensitive word detection request sent by a user terminal is received, wherein the sensitive word detection request includes input text data to be detected.

[0077] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for convenience of understanding and are not used to limit the present application.

[0078] In an embodiment of the present application, the text to be matched is obtained through a sensitive word detection request sent by a user terminal.

[0079] In step S202, the Chinese character Unicode encoding white list is called, and the interference character removal operation is performed on the input text data according to the Chinese character Unicode encoding white list to obtain pure text data.

[0080] In an embodiment of the present application, text data is scanned and input, and interference characters in the text are removed using a Chinese character Unicode encoding whitelist.

[0081] In step S203, a character merging operation is performed on the clean text data to obtain text data to be matched.

[0082] In an embodiment of the present application, clean text data is scanned, and a character merging operation is performed on multiple characters in the text that can be merged into one character.

[0083] In step S204, the sensitive word Trie tree is called, and the root node character of the sensitive word Trie tree is used as the starting point to perform sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data.

[0084] In an embodiment of the present application, starting from the root node of the Trie tree, the Trie tree is traversed downward step by step according to each character in the text to be searched, and the original characters in the text and the characters on the Trie tree nodes enter the similarity matching chain for matching until the complete string is found or it is determined that the string does not exist in the Trie tree.

[0085] In step S205, a comprehensive scoring operation is performed on the matching result data to obtain a comprehensive scoring result.

[0086] In the embodiment of the present application, the sensitive word is scored comprehensively based on the length of the sensitive word finally matched, the model used, the result of the model matching, and the weight of the model.

[0087] In step S206, the comprehensive scoring result is output to the user terminal.

[0088] In an embodiment of the present application, a sensitive word detection method is provided, comprising: receiving a sensitive word detection request sent by a user terminal, wherein the sensitive word detection request includes input text data to be detected; calling a Chinese character Unicode encoding whitelist, and performing an interference character removal operation on the input text data according to the Chinese character Unicode encoding whitelist to obtain pure text data; performing a character merging operation on the pure text data to obtain text data to be matched; calling a sensitive word Trie tree, and using the root node character of the sensitive word Trie tree as a starting point to perform a sensitive word similarity matching operation with the characters of the text data to be matched in a similarity matching chain to obtain matching result data; performing a comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result; and outputting the comprehensive scoring result to the user terminal. Compared with the prior art, the present application automatically and efficiently identifies and filters sensitive words in user input text, thereby protecting the security and compliance of the system.

[0089] In some optional implementations of the embodiments of the present application, the above step of performing an interference character removal operation on the input text data according to the Chinese character Unicode encoding whitelist to obtain pure text data specifically includes the following steps:

[0090] Get the target Unicode block corresponding to the input text data in the Chinese character Unicode encoding whitelist;

[0091] Perform interference character matching operation on input text data according to target Unicode block to obtain interference character matching result;

[0092] According to the interference character matching result, the interference characters in the input text data are removed to obtain pure text data.

[0093] In the embodiment of the present application, according to the Chinese character set and the Chinese character Unicode block range, refer to Figure 4 The schematic diagram of the Unicode encoding whitelist is shown to construct the Chinese character Unicode encoding whitelist. The whitelist checking process is as follows:

[0094] 1) Get the Unicode block to which the character belongs;

[0095] 2) Check whether the character belongs to the character set of Chinese characters in the target block.

[0096] In some optional implementations of the embodiments of the present application, the above step of performing a character merging operation on the clean text data to obtain the text data to be matched specifically includes the following steps:

[0097] Call the ligature library and build a hash table based on the ligature library;

[0098] Select two consecutive characters in the clean text data in sequence, and concatenate the consecutive characters into a string;

[0099] Use the string as the key to perform a hash matching operation in the hash table to obtain a hash matching result;

[0100] According to the hash matching result, the corresponding characters in the clean text data are replaced with the corresponding values ​​in the hash table to obtain the text data to be matched.

[0101] In the embodiments of this application, see Figure 5 The schematic diagram of the combined character library shown in the figure uses the open source decomposition dictionary data of the Open Dictionary Network as the basic data, including simplified and traditional Chinese decomposition. In order to reduce the error when merging Chinese characters, the data needs to be manually labeled, and only the character data that can be decomposed into two normal Chinese characters is retained. This data is saved in a file or database and is used for the operation of merging Chinese characters.

[0102] In the embodiment of the present application, each time a string of text is operated, two characters are sequentially taken from the head of the text to match the compound character library data, and the matching continues until the end of the text. Since the data volume of the decomposed character library is large, such a query will result in very poor performance. If the length of the text to be operated is 10, and the number of rows of compound character library data is 10,000, it is necessary to match (10-1)*10,000*2=180,000 characters. Therefore, a hash table is introduced here. After reading the compound character library, character 1 and character 2 are concatenated into a string as the key, and the original characters are used as values ​​and stored in the hash table. Then the matching is performed. Only one traversal is required to scan all the characters in the text.

[0103] In the embodiment of the present application, the hash table construction process may be:

[0104] 1) Read the combined character library;

[0105] 2) Traverse each row of data in the combined character library;

[0106] 3) Concatenate character 1 and character 2 in each row of data into a string as the key, and use the original characters as the value and store them in the hash table.

[0107] In the embodiment of the present application, the text matching process may be:

[0108] 1) Starting from the beginning of the text, take two characters in sequence to match the compound character library data, and continue matching until the end of the text.

[0109] 2) Concatenate two characters into a string.

[0110] 3) Use the concatenated string as the key to search the hash table.

[0111] 4) If the value exists in the hash table, replace the two characters in the text with the corresponding value in the hash table.

[0112] In some optional implementations of the embodiments of the present application, before the step of performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain with the root node characters of the sensitive word Trie tree as the starting point and obtaining the matching result data, the following steps are also included:

[0113] Constructing an original similarity matching chain, and performing an initialization operation on the original similarity matching chain to obtain an initialized similarity matching chain;

[0114] Construct an original character precise matching model, and add the original character precise matching model to the initial similarity matching chain, where the type of the original character precise matching model is 1 and the binary value is 00000001;

[0115] Read the data in the correction character library, build a correction character precise matching model according to the data in the correction character library, and add the correction character precise matching model to the initialization similarity matching chain, where the type of the correction character precise matching model is 4, and the binary number is 00000100;

[0116] Read the data in the Zhengma character library, build a Zhengma character similarity matching model according to the data in the Zhengma character library, and add the Zhengma character similarity matching model to the initialization similarity matching chain, where the type of the Zhengma character similarity matching model is 8, and the binary is 00001000;

[0117] Read the data in the Cangjie character library, build a Cangjie character similarity matching model according to the data in the Cangjie character library, and add the Cangjie character similarity matching model to the initialization similarity matching chain, where the type of the Cangjie character similarity matching model is 16, and the binary is 00010000;

[0118] Read the data in the Wubi character library, build a Wubi character similarity matching model according to the data in the Wubi character library, and add the Wubi character similarity matching model to the initialized similarity matching chain to obtain a similarity matching chain, wherein the type of the Wubi character similarity matching model is 32, and the binary number is 00100000.

[0119] In the embodiments of this application, see Figure 6 The flowchart of building a similarity matching chain is shown in FIG. The models in the similarity matching chain have different types. For ease of operation, different bits of 1 byte are used to represent different types. When matching characters later, only bitwise AND operations are needed to determine which models participate in the matching of the character. The models in the similarity matching chain are divided into precise matching models and similarity matching models. The precise matching model can only set weights. In addition to setting weights, the similarity matching model can also set similarity algorithms and thresholds. These algorithms include: Jaccard (Jaccard similarity), Sorensen Dice (Sorensen-Dice coefficient), Levenshtein (edit distance), Hamming (Hamming distance), COS (cosine distance). The threshold refers to the similarity that is considered a match. The weight principle of different models: original character matching model > corrected character matching model > other models.

[0120] In the embodiment of the present application, the similarity matching chain construction process may be:

[0121] 1) Initialize the similarity matching chain;

[0122] 2) Build an original character accurate matching model, set the model weight, the model type is 1, the binary is 00000001, and add it to the matching chain;

[0123] 3) Read the data in the correction character library, build a correction character accurate matching model, set the model weight, the model type is 2, the binary is 00000010, and add it to the matching chain;

[0124] 4) Read the data in the Pinyin character library, build a Pinyin character accurate matching model, set the model weight, the model type is 4, the binary is 00000100, and add it to the matching chain;

[0125] 5) Read the data in the Zhengma character library, build a Zhengma character similarity matching model, set the model weight, similarity algorithm and threshold, the model type is 8, the binary is 00001000, and add it to the matching chain;

[0126] 6) Read the data in the Cangjie character library, build a Cangjie character similarity matching model, set the model weight, similarity algorithm and threshold, the model type is 16, the binary is 00010000, and add it to the matching chain;

[0127] Read the data in the Wubi character library, build a Wubi character similarity matching model, set the model weight, similarity algorithm and threshold, the model type is 32, the binary is 00100000, and add it to the matching chain.

[0128] In some optional implementations of the embodiments of the present application, before the step of performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain with the root node characters of the sensitive word Trie tree as the starting point and obtaining the matching result data, the following steps are also included:

[0129] Initialize a root node, where the root node does not contain specific characters;

[0130] Get the sensitive word string to be inserted, and insert the sensitive word string to be inserted character by character starting from the root node;

[0131] When the characters of the sensitive word strings to be inserted are all inserted, the sensitive word Trie tree is obtained.

[0132] In some optional implementations of the embodiments of the present application, the above step of obtaining the sensitive word string to be inserted, and inserting the sensitive word string to be inserted character by character starting from the root node, specifically includes the following steps:

[0133] Determine whether there is a repeated child node corresponding to the character to be inserted in the current node;

[0134] If there is no duplicate child node, create a new child node, link the new child node to the current node, record the character to be inserted, and set the matching model of the character to be inserted to the matching model corresponding to the sensitive word;

[0135] If there is a repeated child node, move to the repeated child node, perform an "OR" operation on the matching model of the character to be inserted and the matching model corresponding to the sensitive word, and set the operation result as the matching model of the character to be inserted.

[0136] In the embodiments of this application, see Figure 7 The diagram of the sensitive word library is shown in the figure. Each row in the sensitive word library contains a sensitive word and a matching model. The calculation method of the matching model value in the word library is to perform an "or" operation on the values ​​of all matching models in the matching chain that the sensitive word wants to use. For example, for example:

[0137] ① Hope to use all models: 1|2|4|8|16|32=63;

[0138] ②Want to use the original character and the correction model: 1|2=3;

[0139] ③ Hope to use the original character, correction, and pinyin model: 1|2|4=7.

[0140] In the embodiment of the present application, the steps of constructing the sensitive word Trie model tree may be:

[0141] 1) Initialization: When constructing a Trie model tree, you first need to initialize a root node. The root node does not contain specific characters, but serves as the starting point for all string insertions;

[0142] 2) Insert sensitive word string:

[0143] 2.1) For each sensitive word string to be inserted, start from the root node and process each character one by one;

[0144] 2.2) For the current character in the sensitive word string, check whether there is a child node of the character under the current node;

[0145] 2.2.1) If it does not exist, create a new child node and link it to the current node, record the character, and set the current matching model of the character to the matching model corresponding to the sensitive word;

[0146] 2.2.2) If it exists, move to the child node corresponding to the character, and perform an “OR” operation on the matching model of the character and the matching model corresponding to the sensitive word, and set the operation result as the current matching model of the character;

[0147] 2.3) Repeat the above steps until all characters in the string have been processed;

[0148] 2.4) Set a flag (such as the end flag) at the node corresponding to the last character of the string to indicate that the node is the end of the string.

[0149] In some optional implementations of the embodiments of the present application, the above-mentioned step of taking the root node character of the sensitive word Trie tree as the starting point and performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data specifically includes the following steps:

[0150] Set the current node as the root node;

[0151] Traverse each character of the text data to be matched from the head, and match the nodes of the sensitive word Trie tree from the two dimensions of depth and width;

[0152] If each character of the text data to be matched is traversed successfully and the end node of the matching sensitive word is reached, it means that the matching sensitive word is successfully matched with the text data to be matched, and the target sensitive word is obtained;

[0153] If no matching characters or nodes can be found during the traversal process, or if the end node of any sensitive word is not reached after traversing the text data to be matched, it means that the match fails and there is no target sensitive word;

[0154] Record the matched target sensitive words and obtain the matching result data.

[0155] In the embodiments of this application, see Figure 8 The flowchart of similarity matching of sensitive words in text is shown in the figure. The key algorithm in the matching process can be:

[0156] Traverse the text to be matched from the beginning, initialize the character position to 0, and increment it until it reaches the end of the text. During the movement, mark the current position as P and continue to execute the following process:

[0157] 1) Initialization: Set the current node as the root node;

[0158] 2) Traverse the string: read each character in the text starting from P in turn, then match the nodes on the Trie tree from the depth and width dimensions, and initialize an empty list to store the matched similar characters;

[0159] 2.1) For the current character, try to match all child nodes of the current node and check whether there is a matching character in the child nodes of the current node;

[0160] 2.1.1) The current character of the text, the character on the child node, and the similarity matching model that the character on the child node wants to use are used as parameters to enter the similarity matching chain for similarity matching;

[0161] 2.1.2) Any model type on the similarity matching chain and the similarity matching model that the character on the child node wants to use are operated on the two values. If the result of the operation is equal to the value of the current model type on the matching chain, the similarity algorithm of the current model is used to calculate the similarity. If it passes the threshold set by the current model, it means that the match is successful, and the next model in the matching chain is entered to continue this process;

[0162] 2.1.3) Save all successfully matched model types and results;

[0163] 2.2) If there are matching characters, copy the old matching list, generate a new matching list, and record the characters on the child node. Then, use the new matching list and the current child node to recursively try to match all the remaining characters starting from the P+1 position of the text and the child node of the current child node.

[0164] 3) Matching success or failure:

[0165] 3.1) If the entire text is traversed successfully and the end node of a sensitive word is reached, it means that the sensitive word matches the text successfully;

[0166] 3.2) If no matching characters or nodes can be found during the traversal process, or if the end node of any sensitive word is not reached after traversing the text, the match fails.

[0167] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0168] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0170] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0171] Further references Fig. 9 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a sensitive word detection device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0172] like Fig. 9 As shown, the sensitive word detection device 200 of the embodiment of the present application includes:

[0173] The request acquisition module 210 is used to receive a sensitive word detection request sent by a user terminal, wherein the sensitive word detection request includes input text data to be detected;

[0174] The interference character removal module 220 is used to call the Chinese character Unicode encoding white list, and perform interference character removal operation on the input text data according to the Chinese character Unicode encoding white list to obtain pure text data;

[0175] A character merging module 230 is used to perform a character merging operation on the clean text data to obtain text data to be matched;

[0176] The sensitive word similarity matching module 240 is used to call the sensitive word Trie tree, and use the root node character of the sensitive word Trie tree as the starting point to perform sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data;

[0177] A comprehensive scoring module 250 is used to perform a comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result;

[0178] The result output module 260 is used to output the comprehensive scoring result to the user terminal.

[0179] In an embodiment of the present application, a sensitive word detection device 200 is provided, comprising: a request acquisition module 210, for receiving a sensitive word detection request sent by a user terminal, wherein the sensitive word detection request includes input text data to be detected; an interference character removal module 220, for calling a Chinese character Unicode encoding whitelist, and performing an interference character removal operation on the input text data according to the Chinese character Unicode encoding whitelist to obtain pure text data; a character merging module 230, for performing a character merging operation on the pure text data to obtain text data to be matched; a sensitive word similarity matching module 240, for calling a sensitive word Trie tree, and taking the root node character of the sensitive word Trie tree as the starting point, respectively performing a sensitive word similarity matching operation with the characters of the text data to be matched in a similarity matching chain to obtain matching result data; a comprehensive scoring module 250, for performing a comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result; and a result output module 260, for outputting the comprehensive scoring result to the user terminal. Compared with the prior art, the present application automatically and efficiently identifies and filters sensitive words in user input text, thereby protecting the security and compliance of the system.

[0180] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0181] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0182] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0183] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the sensitive word detection method. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0184] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as computer-readable instructions for running the sensitive word detection method.

[0185] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0186] The computer device provided by the present application automatically and efficiently identifies and filters sensitive words in user input text, thereby protecting the security and compliance of the system.

[0187] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the sensitive word detection method as described above.

[0188] The computer-readable storage medium provided in the present application automatically and efficiently identifies and filters sensitive words in user input text, thereby protecting the security and compliance of the system.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0190] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A sensitive word detection method, characterized in that: The steps include: Receiving a sensitive word detection request sent by a user terminal, wherein the sensitive word detection request includes input text data to be detected; Calling a Chinese character Unicode encoding whitelist, and performing an interference character removal operation on the input text data according to the Chinese character Unicode encoding whitelist to obtain pure text data; Performing a character merging operation on the clean text data to obtain text data to be matched; Calling the sensitive word Trie tree, and taking the root node character of the sensitive word Trie tree as the starting point, respectively performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain, to obtain matching result data; Performing a comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result; Output the comprehensive scoring result to the user terminal.

2. The sensitive word detection method according to claim 1, characterized in that: The step of performing an interference character removal operation on the input text data according to the Chinese character Unicode encoding white list to obtain pure text data specifically includes the following steps: Obtaining a target Unicode block corresponding to the input text data in the Chinese character Unicode encoding whitelist; Performing an interference character matching operation on the input text data according to the target Unicode block to obtain an interference character matching result; The interference characters in the input text data are removed according to the interference character matching result to obtain the pure text data.

3. The sensitive word detection method according to claim 1, characterized in that: The step of performing a character merging operation on the clean text data to obtain the text data to be matched specifically includes the following steps: Calling a ligature library, and constructing a hash table according to the ligature library; Selecting two consecutive characters in the clean text data in sequence, and concatenating the consecutive characters into a character string; Performing a hash matching operation in the hash table using the character string as a key to obtain a hash matching result; According to the hash matching result, the corresponding characters in the clean text data are replaced with the corresponding values ​​in the hash table to obtain the text data to be matched.

4. The sensitive word detection method according to claim 1, characterized in that: Before the step of taking the root node character of the sensitive word Trie tree as the starting point and performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data, the following steps are also included: Constructing an original similarity matching chain, and performing an initialization operation on the original similarity matching chain to obtain an initialized similarity matching chain; Constructing an original character precise matching model, and adding the original character precise matching model to the initialized similarity matching chain, wherein the type of the original character precise matching model is 1 and the binary value is 00000001; Read the data in the correction character library, build a correction character precise matching model according to the data in the correction character library, and add the correction character precise matching model to the initialization similarity matching chain, wherein the type of the correction character precise matching model is 4, and the binary number is 00000100; Read the data in the Zhengma character library, construct a Zhengma character similarity matching model according to the data in the Zhengma character library, and add the Zhengma character similarity matching model to the initialized similarity matching chain, wherein the type of the Zhengma character similarity matching model is 8, and the binary is 00001000; Read the data in the Cangjie character library, construct a Cangjie character similarity matching model according to the data in the Cangjie character library, and add the Cangjie character similarity matching model to the initialized similarity matching chain, wherein the type of the Cangjie character similarity matching model is 16, which is 00010000 in binary; Read data in the Wubi character library, construct a Wubi character similarity matching model based on the data in the Wubi character library, and add the Wubi character similarity matching model to the initialized similarity matching chain to obtain the similarity matching chain, wherein the type of the Wubi character similarity matching model is 32, and the binary number is 00100000.

5. The sensitive word detection method according to claim 1, characterized in that: Before the step of taking the root node character of the sensitive word Trie tree as the starting point and performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data, the following steps are also included: Initialize a root node, wherein the root node does not contain specific characters; Obtain a sensitive word string to be inserted, and insert the sensitive word string to be inserted character by character starting from the root node; When the characters of the sensitive word character strings to be inserted are all inserted, the sensitive word Trie tree is obtained.

6. The sensitive word detection method according to claim 1, characterized in that: The step of obtaining the sensitive word string to be inserted, and inserting the sensitive word string to be inserted character by character starting from the root node, specifically comprises the following steps: Determine whether there is a repeated child node corresponding to the character to be inserted in the current node; If the repeated child node does not exist, create a new child node, link the new child node to the current node, record the character to be inserted, and set the matching model of the character to be inserted to the matching model corresponding to the sensitive word; If the repeated sub-node exists, move to the repeated sub-node, perform an "OR" operation with the matching model of the character to be inserted and the matching model corresponding to the sensitive word, and set the operation result as the matching model of the character to be inserted.

7. The sensitive word detection method according to claim 1, characterized in that: The step of taking the root node character of the sensitive word Trie tree as the starting point and performing sensitive word similarity matching operations with the characters of the text data to be matched in the similarity matching chain to obtain matching result data specifically includes the following steps: Set the current node as the root node; Traverse each character of the text data to be matched from the head, and match the nodes of the sensitive word Trie tree from two dimensions of depth and width; If each character of the to-be-matched text data is traversed successfully and the end node of the matching sensitive word is reached, it means that the matching sensitive word is successfully matched with the to-be-matched text data, and the target sensitive word is obtained; If no matching characters or nodes can be found during the traversal process, or if the end node of any sensitive word is not reached after traversing the text data to be matched, it means that the match fails and there is no target sensitive word; The matched target sensitive words are recorded to obtain the matching result data.

8. A sensitive word detection device, characterized in that: include: A request acquisition module, configured to receive a sensitive word detection request sent by a user terminal, wherein the sensitive word detection request includes input text data to be detected; An interference character removal module is used to call a Chinese character Unicode encoding white list, and perform an interference character removal operation on the input text data according to the Chinese character Unicode encoding white list to obtain pure text data; A character merging module, used for performing a character merging operation on the clean text data to obtain text data to be matched; A sensitive word similarity matching module is used to call a sensitive word Trie tree, and use the root node character of the sensitive word Trie tree as a starting point to perform sensitive word similarity matching operations with the characters of the text data to be matched in a similarity matching chain to obtain matching result data; A comprehensive scoring module is used to perform a comprehensive scoring operation on the matching result data to obtain a comprehensive scoring result; A result output module is used to output the comprehensive scoring result to the user terminal.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the sensitive word detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the sensitive word detection method according to any one of claims 1 to 7 are implemented.

Images