Sensitive word filtering method and system
By determining the finite automata algorithm to build a sensitive word tree and combining the pinyin file to process multi-tone characters, the problems of high maintenance cost of sensitive word filtering and poor query performance in the existing technology are solved, and efficient and accurate sensitive word filtering is achieved.
Patent Information
- Application Number
- CN202411957705.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-04-18
AI Technical Summary
The existing sensitive word filtering technology is costly to maintain when facing complex and changing sensitive lexicons, making it difficult to deal with complex sensitive words, and has poor query performance, especially when large-scale concurrent requests.
A sensitive word tree is constructed using a definite fertile automata algorithm, and the state transfer information is stored through the HashMap data structure to achieve efficient sensitive word filtering. At the same time, multi-sounding and homophones are processed using pinyin files, the lexicon library is updated in real time, and state transfer is accelerated through hash tables to optimize performance.
It realizes efficient search and filtering of sensitive words, reduces maintenance costs, improves query performance, and can deal with complex sensitive words and large-scale concurrent requests.
Smart Images
Figure HDA0005216318890000011 
Figure HDA0005216318890000021 
Figure HDA0005216318890000022
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a method and system for filtering sensitive words. Background Art
[0002] Common sensitive word filtering methods include using regular expressions to filter sensitive words and traditional SQL queries after sensitive words are stored in the database. Both methods are widely used in sensitive word filtering systems, but each has its limitations.
[0003] The regular expression method has a high maintenance cost when faced with a complex and ever-changing sensitive word library. Once the structure of the sensitive word library changes, the regular expression needs to be rechecked and updated. For complex or changing sensitive word patterns, the writing and maintenance of regular expressions becomes very difficult.
[0004] However, traditional SQL query methods will cause significant performance degradation when there are a large number of sensitive words, and are difficult to adapt to large-scale concurrent requests. Summary of the invention
[0005] In order to overcome the problems in the prior art of high maintenance cost of sensitive word filtering, difficulty in dealing with complex sensitive words, and poor query performance, the purpose of the present invention is to provide a method and system for sensitive word filtering that can meet the needs of efficient search of all sensitive words, has low maintenance cost, and can deal with complex sensitive words.
[0006] The present invention is implemented by the following scheme:
[0007] A method for filtering sensitive words, the method steps are as follows:
[0008] Step 1: Build a sensitive word library;
[0009] Step 2: Filter sensitive words in the input string according to the sensitive word library;
[0010] Step 3: Update sensitive word library in real time;
[0011] Step 4: Optimize sensitive word filtering performance.
[0012] Furthermore, step 1 is specifically as follows:
[0013] Step 1.1: Collect sensitive words;
[0014] Step 1.2: Use the HashMap data structure to store sensitive words for state transfer of the finite automaton. Each character corresponds to a HashMap, which indicates the next state to be transferred from the current state through the character.
[0015] Step 1.3: Create an empty deterministic finite automaton model, that is, an empty root node; and traverse the sensitive word library and add each sensitive word to the deterministic finite automaton model; during the addition process, build a sensitive word tree, starting from the root node, and create or update nodes level by level. If the character exists at the current level, enter the next level; if not, create a new level and node; add an "isEnd" mark to the node of the last character of the sensitive word, indicating that the node is the end of a sensitive word;
[0016] Step 1.4: Create a pinyin mapping table for each Chinese character. During the matching process, the Chinese characters in the input string are converted into pinyin, and homophonic characters are matched according to the pinyin mapping table to handle polyphonic characters and homophonic characters.
[0017] Furthermore, step 2 is specifically as follows:
[0018] Step 2.1: Before filtering sensitive words, pre-process the input string. For the English part of the string, convert it to lowercase; for the Chinese part of the string, convert it to pinyin;
[0019] Step 2.2: Starting from the root node of the sensitive word tree, the sensitive word tree is traversed step by step according to the input characters. Based on the sensitive word tree, the text to be detected is traversed, and the finite automaton model is used to detect whether there are sensitive words. During the detection process, a pointer is initialized to point to the beginning of the text, and each character in the text is traversed. Starting from determining the initial state of the finite automaton, the next state is gradually transferred according to the state transition diagram of the character. In each state, check whether there is an "isEnd" mark. If it exists, it means that a sensitive word has been detected and needs to be replaced or blocked.
[0020] Step 2.3: After converting the Chinese characters in the input string into pinyin, perform homophonic character matching based on the polyphonic character information in the pinyin file. If a homophonic sensitive word is matched, it will be shielded. When a matching node is encountered, check whether the node is the end of a sensitive word, that is, check whether there is an "isEnd" mark. If there is an "isEnd" mark, it means that it is the end of a sensitive word, then reverse search for the parent node and shield the entire sensitive word.
[0021] Step 2.4: Output the filtered string.
[0022] Furthermore, step 3 is specifically as follows: the vocabulary is updated by network loading, and the local vocabulary version number is sent to the server each time a network request is made. If the version number on the server changes, the vocabulary is updated; in addition, the time interval for regularly updating the vocabulary is configured to ensure that the vocabulary can be updated in time and adapt to new sensitive words.
[0023] Furthermore, step 4 is specifically as follows:
[0024] Use hash tables to speed up state transfer: By storing state transfer information in hash tables, the next state can be found quickly, reducing unnecessary state storage and search time;
[0025] Preprocessing sensitive word library: Before building the deterministic finite automaton model, preprocess the sensitive word library.
[0026] A system for filtering sensitive words, the system comprising: a word library construction module, a sensitive word filtering module, a word library update module and a filtering optimization module;
[0027] The word library construction module is used to construct a sensitive word library;
[0028] The sensitive word filtering module is used to filter sensitive words in the input string according to the sensitive word library;
[0029] The word library update module is used to update the sensitive word library in real time;
[0030] The filtering optimization module is used to optimize the sensitive word filtering performance.
[0031] Furthermore, the vocabulary building module is specifically as follows: collecting sensitive words;
[0032] Use HashMap data structure to store sensitive words for state transfer of finite automata. Each character corresponds to a HashMap, indicating the next state that can be transferred from the current state through the character.
[0033] Create an empty deterministic finite automaton model, that is, an empty root node; and traverse the sensitive word library, adding each sensitive word to the deterministic finite automaton model; during the adding process, build a sensitive word tree, starting from the root node, and create or update nodes level by level. If the character exists at the current level, go to the next level; if not, create a new level and node; add an "isEnd" mark to the node of the last character of the sensitive word, indicating that the node is the end of a sensitive word;
[0034] A pinyin mapping table is created for each Chinese character. During the matching process, the Chinese characters in the input string are converted into pinyin, and homophonic characters are matched according to the pinyin mapping table to handle polyphonic characters and homophonic characters.
[0035] Furthermore, the sensitive word filtering module is specifically:
[0036] Before filtering sensitive words, the input string is preprocessed. For the English part of the string, it is converted to lowercase; for the Chinese part of the string, it is converted to pinyin;
[0037] Starting from the root node of the sensitive word tree, the sensitive word tree is traversed step by step according to the input characters. Based on the sensitive word tree, the text to be detected is traversed, and the finite automaton model is used to detect whether there are sensitive words. During the detection process, a pointer is initialized to point to the beginning of the text, and each character in the text is traversed. Starting from determining the initial state of the finite automaton, the next state is gradually transferred according to the state transition diagram of the character. In each state, check whether there is an "isEnd" mark. If it exists, it means that a sensitive word has been detected and needs to be replaced or blocked.
[0038] After converting the Chinese characters in the input string into pinyin, the homophonic characters are matched according to the polyphonic character information in the pinyin file. If a homophonic sensitive word is matched, it is shielded. When a matching node is encountered, check whether the node is the end of a sensitive word, that is, check whether there is an "isEnd" mark. If there is an "isEnd" mark, it means that it is the end of a sensitive word, then reverse search for the parent node and shield the entire sensitive word.
[0039] Output the filtered string.
[0040] Furthermore, the vocabulary update module specifically updates the vocabulary by network loading, and sends the local vocabulary version number to the server each time a network request is made. If the version number on the server changes, the vocabulary is updated. In addition, the time interval for scheduled vocabulary updates is configured to ensure that the vocabulary can be updated in a timely manner and adapt to new sensitive words.
[0041] Furthermore, the filtering optimization module is specifically as follows:
[0042] Use hash tables to speed up state transfer: By storing state transfer information in hash tables, the next state can be found quickly, reducing unnecessary state storage and search time;
[0043] Preprocessing sensitive word library: Before building the deterministic finite automaton model, preprocess the sensitive word library.
[0044] The beneficial effects of the present invention are:
[0045] The present invention provides a method and system for filtering sensitive words, which adopts a deterministic finite automaton algorithm to implement sensitive word filtering. The deterministic finite automaton algorithm is a pattern matching technology based on state transition. It efficiently identifies and matches specific patterns in an input string by constructing a finite state machine and a deterministic transfer function. The deterministic finite automaton algorithm constructs sensitive words into a sensitive word tree, thereby realizing efficient search for all sensitive words by traversing the input string once. At the same time, the accuracy and flexibility of sensitive word filtering are further enhanced by combining pinyin files to process homophonic characters. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 is a flow chart of the method of the present invention;
[0047] Figure 2 It is a structural block diagram of the system of the present invention;
[0048] Figure 3 This is a schematic diagram of the sensitive word tree. DETAILED DESCRIPTION
[0049] The present invention will be further described below in conjunction with the accompanying drawings.
[0050] See also Figure 1 , a method for filtering sensitive words, the method steps are as follows:
[0051] Step 1: Build a sensitive word library;
[0052] Step 2: Filter sensitive words in the input string according to the sensitive word library;
[0053] Step 3: Update sensitive word library in real time;
[0054] Step 4: Optimize sensitive word filtering performance.
[0055] The present invention will be further described below in conjunction with a specific embodiment:
[0056] A method for filtering sensitive words, the method comprising the following steps:
[0057] Step 1: Construction of sensitive word library;
[0058] Step 1.1: Collect sensitive words;
[0059] In order to build a comprehensive and accurate sensitive word library, sensitive words are collected from multiple channels such as the Internet and the media, and strictly screened and classified. These sensitive words should cover all kinds of sensitive words to ensure detection capabilities.
[0060] Step 1.2: Design data structure;
[0061] HashMap data structure is used to store sensitive words for state transfer of deterministic finite automata. Each character of a sensitive word corresponds to a HashMap, which indicates the next state that can be transferred from the current state through the character, making the state transfer process more efficient and flexible.
[0062] The concept of "transferring from the current state to the next state through this character" means:
[0063] Current state: Determine the current state of the finite automaton. In sensitive word detection, each state can represent the length of the prefix of the sensitive word that has been matched. For example, if the sensitive word is "badword", then state 0 means that no characters have been matched, state 1 means that "b" has been matched, state 2 means that "ba" has been matched, and so on.
[0064] Character: The character currently input. For example, if the current input is "b", and the finite automaton is determined to be in state 0, then we need to find the next state to which we can transfer from state 0 via the character 'b'.
[0065] The next state to be transferred to: Based on the current state and the input character, determine the new state that the finite automaton will transfer to. This new state can be a part of the continued matching of the sensitive word (for example, from state 0 to state 1 through the character 'b'), or it can be a special state indicating the end of the match or reset (for example, if the complete sensitive word is matched, it may transfer to a state indicating "match successful", or if the input character does not match the prefix of the current sensitive word, it may transfer to a "reset" state).
[0066] In a HashMap, each state can correspond to a HashMap, where the key is a character (or an integer representation of a character) and the value is the identifier of the next state to be transferred from the current state through the character.
[0067] Example
[0068] Suppose we have the following list of sensitive words: ["bad", "word"]
[0069] We can construct a deterministic finite automaton as follows:
[0070] State 0: Initial state, no characters are matched.
[0071] Status 1: "b" has been matched (corresponding to the prefix of the sensitive word "bad").
[0072] State 2: "ba" has been matched (corresponding to the prefix of the sensitive word "bad").
[0073] State 3: The complete sensitive word "bad" is matched (this is a terminal state, indicating that a sensitive word is detected).
[0074] State 4: "w" has been matched (corresponding to the prefix of the sensitive word "word").
[0075] State 5: "wo" has been matched (corresponding to the prefix of the sensitive word "word").
[0076] Status 6: "wor" has been matched (corresponding to the prefix of the sensitive word "word").
[0077] State 7: The complete sensitive word "word" is matched (this is another terminal state).
[0078] Step 1.3: Initialize the sensitive word library;
[0079] Create an empty deterministic finite automaton model, that is, an empty root node; and traverse the sensitive word library, adding each sensitive word to the deterministic finite automaton model; during the adding process, build a sensitive word tree, starting from the root node,
[0080] Create or update nodes level by level. If the character exists at the current level, go to the next level. If not, create a new level and node. Add an "isEnd" tag to the node of the last character of a sensitive word, indicating that the node is the end of a sensitive word.
[0081] Deterministic finite automata are a prior art and are mathematical models used to represent and process strings over a finite character set (or language).
[0082] It consists of the following four components:
[0083] State set (Q): contains a finite number of states, including an initial state and one or more accepting states.
[0084] Input alphabet (Σ): Contains the set of characters that can be read by the deterministic finite automaton.
[0085] Transition function (δ): A function that defines the transition from one state to another, based on the current state and the input character.
[0086] Initial state (q0): determines the state of the finite automaton at the beginning.
[0087] Accepting state set (F): contains all final states accepted by the deterministic finite automaton.
[0088] An empty DFA means that the basic skeleton of the DFA has been set up, but nothing has been filled in yet. The empty root node refers to the starting point of this structure, where you start adding states and transitions.
[0089] Waiting to add specific status and transition rules
[0090] Step 1.4: Processing polyphones and homophones;
[0091] In order to handle the problem of polyphones and homophones, the pinyin file is loaded, and a pinyin mapping table is created for each Chinese character to associate the different pronunciations of polyphones. A pinyin mapping table is created for each Chinese character. During the matching process, the Chinese characters in the input string are converted to pinyin, and homophones are matched according to the pinyin mapping table. In this way, even if the Chinese characters in the input string and the Chinese characters in the sensitive words have different pronunciations but the same pronunciation, they can be correctly identified and processed.
[0092] Step 2: Filter sensitive words;
[0093] Step 2.1: Preprocess the input string;
[0094] Before filtering sensitive words, the input string is preprocessed. The English part of the string is converted to lowercase; the Chinese part of the string is converted to pinyin. This process ensures that the sensitivity of uppercase and lowercase letters and Chinese pinyin remains consistent.
[0095] Step 2.2: Traverse the sensitive word tree;
[0096] Starting from the root node of the sensitive word tree, the sensitive word tree is traversed step by step according to the input characters. Based on the sensitive word tree, the text to be detected is traversed, and the finite automaton model is used to detect whether there are sensitive words. During the detection process, a pointer is initialized to point to the beginning of the text, and each character in the text is traversed. Starting from determining the initial state of the finite automaton, the next state is gradually transferred according to the state transition diagram of the character. In each state, check whether there is an "isEnd" mark. If it exists, it means that a sensitive word has been detected and needs to be replaced or blocked.
[0097] Step 2.3: homophonic word processing;
[0098] After converting the Chinese characters in the input string into pinyin, the homophonic characters are matched according to the polyphonic character information in the pinyin file. If a homophonic sensitive word is matched, it is shielded. When a matching node is encountered, check whether the node is the end of a sensitive word, that is, check whether there is an "isEnd" mark. If there is an "isEnd" mark, it means that it is the end of a sensitive word, then reverse search for the parent node and shield the entire sensitive word.
[0099] Step 2.4: Output the filtered string. The sensitive words in the string have been replaced or blocked, meeting the requirements.
[0100] Step 3: Set up the vocabulary update mechanism;
[0101] In order to ensure the real-time and accuracy of the sensitive vocabulary, the vocabulary is updated by network loading. Each time a network request is made, the local vocabulary version number is sent to the server. If the version number on the server changes, the vocabulary is updated. In addition, the time interval for scheduled vocabulary updates is configured to ensure that the vocabulary can be updated in a timely manner and adapt to new sensitive words.
[0102] Step 4: Optimize performance;
[0103] In order to improve the performance of sensitive word filtering, the following optimization measures are taken:
[0104] Use hash tables to speed up state transfer: By storing state transfer information in a hash table, the next state can be found quickly.
[0105] Reduce unnecessary state storage and lookup time;
[0106] Preprocessing sensitive word library: Before building the deterministic finite automaton model, preprocess the sensitive word library, such as removing duplicate words, merging synonyms, etc. This can reduce the amount of calculation in the detection process and improve detection efficiency.
[0107] In summary, the present invention can achieve efficient, accurate and real-time sensitive word filtering by building a comprehensive and accurate sensitive word library, designing an efficient filtering process, implementing a word library update mechanism and optimizing performance. The present invention can be widely used in various text processing scenarios, such as social media, forums, blogs, etc., to effectively block sensitive words and protect the legitimate rights and interests of users.
[0108] See also Figure 2 , a system for filtering sensitive words, the system comprising: a word library construction module, a sensitive word filtering module, a word library update module and a filtering optimization module;
[0109] The word library construction module is used to construct a sensitive word library;
[0110] The sensitive word filtering module is used to filter sensitive words in the input string according to the sensitive word library;
[0111] The word library update module is used to update the sensitive word library in real time;
[0112] The filtering optimization module is used to optimize the sensitive word filtering performance.
[0113] In one embodiment of the present invention, the word library construction module specifically includes: collecting sensitive words;
[0114] Use HashMap data structure to store sensitive words for state transfer of finite automata. Each character corresponds to a HashMap, indicating the next state that can be transferred from the current state through the character.
[0115] Create an empty deterministic finite automaton model, that is, an empty root node; and traverse the sensitive word library, adding each sensitive word to the deterministic finite automaton model; during the adding process, build a sensitive word tree, starting from the root node, and create or update nodes level by level. If the character exists at the current level, go to the next level; if not, create a new level and node; add an "isEnd" mark to the node of the last character of the sensitive word, indicating that the node is the end of a sensitive word;
[0116] A pinyin mapping table is created for each Chinese character. During the matching process, the Chinese characters in the input string are converted into pinyin, and homophonic characters are matched according to the pinyin mapping table to handle polyphonic characters and homophonic characters.
[0117] In one embodiment of the present invention, the sensitive word filtering module is specifically:
[0118] Before filtering sensitive words, the input string is preprocessed. For the English part of the string, it is converted to lowercase; for the Chinese part of the string, it is converted to pinyin;
[0119] Starting from the root node of the sensitive word tree, the sensitive word tree is traversed step by step according to the input characters. Based on the sensitive word tree, the text to be detected is traversed, and the finite automaton model is used to detect whether there are sensitive words. During the detection process, a pointer is initialized to point to the beginning of the text, and each character in the text is traversed. Starting from determining the initial state of the finite automaton, the next state is gradually transferred according to the state transition diagram of the character. In each state, check whether there is an "isEnd" mark. If it exists, it means that a sensitive word has been detected and needs to be replaced or blocked.
[0120] After converting the Chinese characters in the input string into pinyin, the homophonic characters are matched according to the polyphonic character information in the pinyin file. If a homophonic sensitive word is matched, it is shielded. When a matching node is encountered, check whether the node is the end of a sensitive word, that is, check whether there is an "isEnd" mark. If there is an "isEnd" mark, it means that it is the end of a sensitive word, then reverse search for the parent node and shield the entire sensitive word.
[0121] Output the filtered string.
[0122] In one embodiment of the present invention, the vocabulary update module specifically updates the vocabulary by network loading, and sends the local vocabulary version number to the server each time a network request is made. If the version number on the server changes, the vocabulary is updated. In addition, the time interval for regularly updating the vocabulary is configured to ensure that the vocabulary can be updated in a timely manner and adapt to new sensitive words.
[0123] In one embodiment of the present invention, the filtering optimization module is specifically:
[0124] Use hash tables to speed up state transfer: By storing state transfer information in a hash table, the next state can be found quickly.
[0125] Reduce unnecessary state storage and lookup time;
[0126] Preprocessing sensitive word library: Before building the deterministic finite automaton model, preprocess the sensitive word library.
[0127] The following is one of the specific application scenarios of the present invention:
[0128] A social media platform adopts the sensitive word filtering system of the present invention to achieve accurate review of user-posted content;
[0129] 1. Build a sensitive word library
[0130] First, the platform will collect and organize various sensitive words, which cover multiple aspects, to ensure the comprehensiveness of the review. In order to uniformly handle, all sensitive words will be converted to lowercase to reduce unnecessary complexity in the subsequent matching process.
[0131] Next, the platform will use these sensitive words to build a sensitive word tree. The sensitive word tree is a special data structure that arranges sensitive words according to their character order, so that it can quickly locate and match the input text content. In addition, in order to deal with the problem of polyphones in Chinese, the platform will also load a pinyin file, which contains the pinyin information of polyphones, so as to make more accurate judgments during the matching process.
[0132] For example Figure 3 It is a sensitive word library that is constructed.
[0133] 2. Review of user-posted content
[0134] When a user posts text content, the system automatically reviews it. First, the system converts the text content into lowercase (for English) and pinyin (for Chinese) to match it with the data in the sensitive word library. Then, the system starts from the root node of the sensitive word tree and gradually traverses the sensitive word tree according to the characters entered.
[0135] During the traversal process, if the system encounters a matching node, and the node happens to be the end of a sensitive word, the system will immediately reversely look up the parent node to determine the specific location of the sensitive word in the text. Once the sensitive word is found, the system will block it or replace it with a specific placeholder to ensure the compliance of the content.
[0136] In addition, for the problem of matching homophones in Chinese, the system will process it according to the polyphone information in the pinyin file. If the input text content is similar to a sensitive word in pinyin (i.e. homophonic), the system will also treat it as a sensitive word and block or replace it.
[0137] 3. Dictionary update and maintenance
[0138] In order to ensure the accuracy and real-time performance of the sensitive word filtering system, the platform will regularly update and maintain the sensitive word library. The system will regularly obtain the latest sensitive word library version number from the server and compare it with the local version. If the version number changes, the system will immediately download and update the local sensitive word library to ensure that the data used in the audit process is the latest.
[0139] At the same time, the platform will also configure a time interval for updating the word library to ensure the real-time and accuracy of the word library. This measure can effectively deal with newly emerging sensitive words or outdated sensitive words, thereby improving the efficiency and accuracy of the review.
[0140] In summary, this system provides an efficient and accurate content review mechanism for social media platforms. By building a sensitive word library, reviewing user-posted content, and regularly updating and maintaining the word library, the platform can ensure the compliance and security of user-posted content and provide users with a healthier and safer social environment.
[0141] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.
Claims
1. A method for filtering sensitive words, characterized in that: The method steps are as follows: Step 1: Build a sensitive word library; Step 2: Filter sensitive words in the input string according to the sensitive word library; Step 3: Update sensitive word library in real time; Step 4: Optimize sensitive word filtering performance.
2. A sensitive word filtering method according to claim 1, characterized in that: Step 1 is as follows: Step 1.1: Collect sensitive words; Step 1.2: Use the HashMap data structure to store sensitive words for determining the state transition of the finite automaton to form a state transition diagram. Each character corresponds to a HashMap, indicating the next state to be transferred from the current state through the character. Step 1.3: Create an empty deterministic finite automaton model, that is, an empty root node; and traverse the sensitive word library and add each sensitive word to the deterministic finite automaton model; during the addition process, build a sensitive word tree, starting from the root node, and create or update nodes level by level. If the character exists at the current level, go to the next level; if not, create a new level and node; add an "isEnd" mark to the node of the last character of the sensitive word, indicating that the node is the end of a sensitive word; Step 1.4: Create a pinyin mapping table for each Chinese character. During the matching process, the Chinese characters in the input string are converted into pinyin, and homophonic characters are matched according to the pinyin mapping table to handle polyphonic characters and homophonic characters.
3. A sensitive word filtering method according to claim 2, characterized in that: Step 2 is as follows: Step 2.1: Before filtering sensitive words, pre-process the input string. For the English part of the string, convert it to lowercase; for the Chinese part of the string, convert it to pinyin; Step 2.2: Starting from the root node of the sensitive word tree, the sensitive word tree is traversed step by step according to the input characters. Based on the sensitive word tree, the text to be detected is traversed, and a finite automaton model is used to detect whether a sensitive word exists. During the detection process, a pointer is initialized to point to the beginning of the text, and each character in the text is traversed; starting from the initial state of the finite automaton, it is gradually transferred to the next state according to the state transition diagram of the character; in each state, it is checked whether there is an "isEnd" mark. If it exists, it means that a sensitive word has been detected and needs to be detected. Line replacement or masking processing; Step 2.3: After converting the Chinese characters in the input string into pinyin, perform homophonic character matching based on the polyphonic character information in the pinyin file. If a homophonic sensitive word is matched, it will be shielded. When a matching node is encountered, check whether the node is the end of a sensitive word, that is, check whether there is an "isEnd" mark. If there is an "isEnd" mark, it means that it is the end of a sensitive word, then reversely search for the parent node and shield the entire sensitive word. Step 2.4: Output the filtered string.
4. A sensitive word filtering method according to claim 1, characterized in that: Step 3 is as follows: Update the vocabulary by network loading. Send the local vocabulary version number to the server each time a network request is made. If the version number on the server changes, update the vocabulary. In addition, configure the time interval for updating the vocabulary to ensure that the vocabulary can be updated in time and adapt to new sensitive words.
5. The method for filtering sensitive words according to claim 1, characterized in that: Step 4 is as follows: Use hash tables to speed up state transfer: By storing state transfer information in a hash table, the next state can be found quickly. Reduce unnecessary state storage and lookup time; Preprocessing sensitive word library: Before building the deterministic finite automaton model, preprocess the sensitive word library.
6. A system for filtering sensitive words, characterized in that: The system includes: a vocabulary building module, a sensitive word filtering module, a vocabulary updating module and a filtering optimization module; The word library construction module is used to construct a sensitive word library; The sensitive word filtering module is used to filter sensitive words in the input string according to the sensitive word library; The word library update module is used to update the sensitive word library in real time; The filtering optimization module is used to optimize the sensitive word filtering performance.
7. A system for filtering sensitive words according to claim 6, characterized in that: The vocabulary building module is specifically as follows: collecting sensitive words; Use HashMap data structure to store sensitive words for state transfer of finite automata. Each character corresponds to a HashMap, indicating the next state that can be transferred from the current state through the character. Create an empty deterministic finite automaton model, that is, an empty root node; and traverse the sensitive word library and add each sensitive word to the deterministic finite automaton model; During the adding process, a sensitive word tree is constructed, starting from the root node. Create or update nodes level by level. If the character exists at the current level, go to the next level. If not, A new level and node are created; an "isEnd" mark is added to the node of the last character of the sensitive word, indicating that the node is the end of a sensitive word; A pinyin mapping table is created for each Chinese character. During the matching process, the Chinese characters in the input string are converted into pinyin, and homophonic characters are matched according to the pinyin mapping table to handle polyphonic characters and homophonic characters.
8. A system for filtering sensitive words according to claim 6, characterized in that: The sensitive word filtering module is as follows: Before filtering sensitive words, the input string is preprocessed. For the English part of the string, it is converted to lowercase; for the Chinese part of the string, it is converted to pinyin; Starting from the root node of the sensitive word tree, the sensitive word tree is traversed step by step according to the input characters. On the basis of the sensitive word tree, Traverse the text to be tested and detect whether there are sensitive words through the finite automaton model; During the detection process, a pointer is initialized to point to the beginning of the text, and each character in the text is traversed; starting from determining the initial state of the finite automaton, it is gradually transferred to the next state according to the state transition diagram of the character; in each state, it is checked whether there is an "isEnd" mark. If it exists, it means that a sensitive word has been detected and needs to be replaced or blocked; After converting the Chinese characters in the input string into pinyin, the homophonic characters are matched according to the polyphonic character information in the pinyin file. If a homophonic sensitive word is matched, it is shielded. When a matching node is encountered, check whether the node is the end of a sensitive word, that is, check whether there is an "isEnd" mark. If there is an "isEnd" mark, it means that it is the end of a sensitive word, then reverse search for the parent node and shield the entire sensitive word. Output the filtered string.
9. A system for filtering sensitive words according to claim 6, characterized in that: The vocabulary update module specifically updates the vocabulary using network loading. Each time a network request is made, the local vocabulary version number is sent to the server. If the version number on the server changes, the vocabulary is updated. In addition, the time interval for scheduled vocabulary updates is configured to ensure that the vocabulary can be updated in a timely manner and adapt to new sensitive words.
10. A system for filtering sensitive words according to claim 6, characterized in that: The filtering optimization module is as follows: Use hash tables to speed up state transfer: By storing state transfer information in a hash table, the next state can be found quickly. Reduce unnecessary state storage and lookup time; Preprocessing sensitive word library: Before building the deterministic finite automaton model, preprocess the sensitive word library.