Method and system for high performance anti-bypass keyword filtering

Abstract

The application provides a high-performance bypass keyword filtering method and system, and the method comprises the following steps: defining a character type; generating a dictionary tree of all keywords that need to be filtered; and identifying the keywords in the string that need to be detected through the dictionary tree. The application can efficiently detect the string and accurately identify the keywords in the string. Even when other characters are mixed in the string, the application can still identify the keywords in the string and prevent character skipping bypass.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Internet information interaction, and particularly relates to a high-performance keyword filtering method and system. BACKGROUND

[0002] In the Internet, there are many scenarios that need to submit data to the server, such as setting a nickname, publishing a comment, sending a barrage, and posting a forum, etc., so the server needs to identify and process a large amount of data. For a large number of concurrent data processing, the keyword efficient identification speed and system response time are particularly important performance indicators. Filtering keywords is a checking technique for system submitted data, a keyword library is constructed, and then it is judged whether the submitted data can match the keyword. If the keyword can be matched, the submitted data needs to be processed, the matched keyword is shielded, so as to obtain data without keywords and other illegal content, and then the data is published and processed.

[0003] However, the matching method by traversing the keyword library has the problem of low processing efficiency when processing a large amount of data requests. Another method is to use an optimized structured word library that is conducive to fast retrieval, such as a tree structure, and then traverse the submitted data for matching. The content length of the traversed word library is greatly reduced, and the efficiency is improved, but it cannot effectively identify data in which other characters are mixed into the keyword, and it is easy to be bypassed. SUMMARY

[0004] In view of the defects of the prior art, the application provides a high-performance keyword filtering method and system that can effectively solve the above problems.

[0005] The technical scheme adopted by the application is as follows:

[0006] The application provides a high-performance keyword filtering method, comprising the following steps:

[0007] Step 1, defining character types;

[0008] Step 2, generating a dictionary tree for all keywords that need to be filtered; in the dictionary tree, assuming that any keyword is composed of n characters in order, which are: the 0th position character, the 1st position character,..., and the n-1th position character; the representation in the dictionary tree is formed by cascading 2n nodes in order, wherein the 2n nodes represent: the character type of the 0th position character, the 0th position character, the character type of the 1st position character, the 1st position character,..., the character type of the n-1th position character, and the n-1th position character in order.

[0009] Step 3, traversing the string to be detected, if the current position does not exist a character, representing the end of the string, the detection ends; if the current position exists a character, reading the current character char i , according to the character type defined in step 1, obtaining the character type type i of the current character char i , executing step 4;

[0010] Step 4, judging whether the node corresponding to the character type type i can be matched in the dictionary tree, if yes, assuming the matched node as node j , then executing step 5; if no, executing step 8;

[0011] Step 5, setting the detection state as the dictionary detection state, executing step 6;

[0012] Step 6, locating the next directly connected node of the node node j in the dictionary tree, representing as node j+1 , then judging whether the character represented by the node node j+1 matches the current character char i , if yes, recording the current character char i and the position of the current character char i in the string, then executing step 7; if no, setting the detection state as the no detection state, clearing the record, resetting the dictionary tree as the initial state, then returning to step 3, traversing the next character of the string;

[0013] Step 7, setting the detection state as the dictionary detection state; judging whether the node node j+1 has the next directly connected node in the dictionary tree, if no, representing the end of the dictionary tree, the detection ends; if yes, returning to step 3, traversing the next character of the string;

[0014] Step 8, judging the current detection state, if as the dictionary detection state, keeping the current located node in the dictionary tree unchanged, returning to step 3, traversing the next character of the string; if as the no detection state, resetting the dictionary tree as the initial state, then returning to step 3, traversing the next character of the string;

[0015] Through steps 3 to 8, finally recording the characters and their positions in the string, arranging the recorded characters according to their positions in the string, forming a character sequence, which is the detected character sequence to be filtered in the detected string.

[0016] Preferably, the character types include four types, namely, Chinese character type, English letter character type, number character type and other character type.

[0017] Preferably, the English letter character type includes capital English letter character type and small letter English letter character type.

[0018] The application also provides a high-performance system for preventing bypassing keyword filtering, comprising:

[0019] a character type defining module for defining character types;

[0020] a dictionary tree generating module for generating a dictionary tree for all keywords to be filtered; in the dictionary tree, assuming that any keyword is composed of n characters in sequence, namely, 0th position character, 1st position character,..., and (n-1)th position character, the representation of the keyword in the dictionary tree is formed by cascading 2n nodes in front-back sequence, wherein the 2n nodes represent, in front-back sequence, character type of the 0th position character, the 0th position character, character type of the 1st position character, the 1st position character,..., character type of the (n-1)th position character, and the (n-1)th position character.

[0021] a bypass keyword detecting module for identifying keywords from a string to be detected by using the following method:

[0022] Step 3, traversing the string to be detected, if there is no character in the current position, it represents the end of the string and the end of the detection; if there is a character in the current position, reading the current character char i , obtaining character type type i of the current character char i according to the character types defined in step 1, and executing step 4;

[0023] Step 4, judging whether a node corresponding to the character type type i can be matched in the dictionary tree, if yes, assuming that the matched node is node j , then executing step 5; if no, executing step 8;

[0024] Step 5, setting the detection state to a dictionary detection state, and executing step 6;

[0025] Step 6, locating a node directly connected to the next level of node j in the dictionary tree, denoted as node j+1 , and then judging whether the character represented by node j+1 is the same as the current character char iIf a match is found, record the current character (char). i and the current character char i Find the position in the string, and then proceed to step 7; if there is no match, set the detection state to no detection state, clear the record, reset the trie to the initial state, and then return to step 3 to traverse the next character of the string;

[0026] Step 7: Set the detection status to "in progress" in dictionary detection; in the trie, determine the node. j+1 Check if there is a directly connected node at the next level. If not, the trie is complete and the check ends. If there is, return to step 3 and traverse the next character of the string.

[0027] Step 8: Determine the current detection state. If it is in the state of dictionary detection, the current node position in the trie remains unchanged, and return to Step 3 to traverse the next character of the string. If it is in the state of no detection, the trie is reset to the initial state, and then return to Step 3 to traverse the next character of the string.

[0028] Through steps 3 to 8, the characters and their positions in the string are finally recorded. The recorded characters are arranged according to their positions in the string to form a character sequence, which is the character sequence that needs to be filtered in the string being detected.

[0029] The high-performance anti-bypass keyword filtering method and system provided by this invention have the following advantages:

[0030] This invention can efficiently detect strings and accurately identify keywords in strings. In particular, it can still identify keywords in strings when other characters are mixed in, preventing characters from being skipped. Attached Figure Description

[0031] Figure 1 A flowchart illustrating the high-performance keyword filtering prevention method provided by this invention;

[0032] Figure 2 An architecture diagram of the high-performance keyword filtering bypass method provided by the present invention;

[0033] Figure 3 An example diagram of a trie provided by the present invention. Detailed Implementation

[0034] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0035] Reference Figure 1 The present application provides a high-performance anti-bypass keyword filtering method, comprising the following steps:

[0036] Step 1, define character types; for example, there are four character types, namely: Chinese character type, English letter character type, numerical character type and other character type. Among them, the English letter character type includes uppercase English letter character type and lowercase English letter character type.

[0037] Step 2, generate a dictionary tree for all keywords that need to be filtered; in the dictionary tree, assume that any keyword is composed of n characters in order, which are: 0th position character, 1st position character, …, n-1th position character; Its representation in the dictionary tree is: 2n nodes are concatenated in front and back order, wherein 2n nodes represent: character type of 0th position character, 0th position character, character type of 1st position character, 1st position character, …, character type of n-1th position character, n-1th position character in front and back order respectively;

[0038] Step 3, traverse the string to be detected, if the current position does not exist character, it represents the end of the traversal string, and the detection ends; if the current position exists character, read the current character char i , according to the character type defined in step 1, obtain the character type type i of the current character char i , execute step 4;

[0039] Step 4, in the dictionary tree, judge whether the node corresponding to the character type type i can be matched, if yes, assume that the matched node is node j , then execute step 5; if not, execute step 8;

[0040] Step 5, set the detection state to the dictionary detection state, execute step 6;

[0041] Step 6, in the dictionary tree, locate the node directly connected to the next level of node j , denoted as node j+1 , then judge whether the character represented by node j+1 matches the current character char i , if yes, record the current character char i and the current character char iIf matched, the detection state is set to dictionary detection state, the current position of the character in the string is recorded, and step 7 is performed; if not matched, the detection state is set to no detection state, the record is emptied, the dictionary tree is reset to the initial state, and then step 3 is returned to traverse the next character of the string;

[0042] Step 7, the detection state is set to the dictionary detection state; in the dictionary tree, it is judged whether the node node j+1 has a node of the next level direct connection, if not, it indicates that the dictionary tree is completed, and the detection is ended; if yes, step 3 is returned to traverse the next character of the string;

[0043] Step 8, it is judged whether the current detection state is the dictionary detection state, if yes, the current position of the node in the dictionary tree is kept unchanged, step 3 is returned to traverse the next character of the string; if no, the dictionary tree is reset to the initial state, and then step 3 is returned to traverse the next character of the string;

[0044] Through steps 3 to 8, the character and its position in the string are finally recorded, and the characters recorded are arranged according to their positions in the string to form a character sequence, which is the character sequence of the character to be filtered in the string to be detected.

[0045] The application also provides a high-performance keyword filtering system to prevent bypassing.

[0046] The character type definition module is used for defining the character type.

[0047] The dictionary tree generation module is used for generating a dictionary tree of all the keywords to be filtered; in the dictionary tree, it is assumed that any keyword is composed of n characters in sequence, which are respectively: the 0th position character, the 1st position character, …, and the n-1th position character; and the representation in the dictionary tree is that 2n nodes are cascaded in front and back order, wherein the 2n nodes represent respectively: the character type of the 0th position character, the 0th position character, the character type of the 1st position character, the 1st position character, …, the character type of the n-1th position character, and the n-1th position character.

[0048] The anti-bypass keyword detection module is used for identifying the keyword from the string to be detected by using the following method:

[0049] Step 3, the string to be detected is traversed, if there is no character at the current position, it indicates that the string traversal is ended, and the detection is ended; if there is a character at the current position, the current character char i is read, the character type type i of the current character char i is obtained according to the character type defined in step 1, and step 4 is performed.

[0050] Step 4, in the dictionary tree, judge whether a node matching to the character type type i corresponding node, if yes, assume the matching node as node j , then execute step 5; if no, execute step 8;

[0051] Step 5, set the detection state as the dictionary detection state, execute step 6;

[0052] Step 6, in the dictionary tree, locate the node node j directly connected to the next level, denoted as node j+1 , then judge whether the character represented by node j+1 matches the current character char i , if yes, record the current character char i and the position of the current character char i in the string, then execute step 7; if no, set the detection state as the no detection state, clear the record, reset the dictionary tree to the initial state, then return to step 3 to traverse the next character of the string;

[0053] Step 7, set the detection state as the dictionary detection state; in the dictionary tree, judge whether node j+1 has a node directly connected to the next level, if no, it means the dictionary tree is complete, the detection is finished; if yes, return to step 3 to traverse the next character of the string;

[0054] Step 8, judge the current detection state, if it is the dictionary detection state, keep the current position in the dictionary tree unchanged, return to step 3 to traverse the next character of the string; if it is the no detection state, reset the dictionary tree to the initial state, then return to step 3 to traverse the next character of the string;

[0055] Through steps 3 to 8, the characters and their positions in the string are finally recorded, and the recorded characters are arranged according to their positions in the string to form a character sequence, which is the character sequence detected in the string to be detected and needs to be filtered.

[0056] An embodiment is listed as follows:

[0057] Step 1, define the character types including four types, which are: Chinese character type, uppercase and lowercase letter type, number type and other type, the four types of characters are represented by numbers 1, 2, 3 and 4 respectively.

[0058] In practical application, four common character types (1: Chinese characters [19968, 40917], 2: capital letters [65, 90], 3: numbers [48, 57], 4: others) can be defined by using Unicode character set, and the character types can be extended.

[0059] Step 2: generating a dictionary tree; as shown in the following figure, an example of a dictionary tree is provided. In this example, "word2 word3 word4 word5 word6" is a keyword to be recognized. Figure 3

[0060] Step 3: assuming that the string to be detected is "word0 word1 word2 word3 word4ab word5 word6".

[0061] Step 4: traversing the first character 'word0' of the string, obtaining its character type 1 (Chinese character), and entering step 5.

[0062] Step 5: character type matching.

[0063] In the dictionary tree, the first-level node with character type 1 (Chinese character) is matched, which is represented as node [word0], and step 6 is entered.

[0064] Step 6: setting the detection state to the dictionary detection state, and executing step 7.

[0065] Step 7: dictionary tree modification and character matching.

[0066] Dictionary tree modification, i.e., locating the next-level directly connected node node [word0+1] of node [word0] in the dictionary tree, and performing character matching, i.e., judging whether the character of node [word0] matches the current character 'word0'. At this time, the matching fails, the detection state is set to the no-detection state, the record is cleared, the dictionary tree is reset to the initial state, and then step 8 is executed.

[0067] Step 8: traversing the second character 'word1' of the string, obtaining its character type 1 (Chinese character), and executing the same way as the first character 'word0'. The character type matching is successful, but the character matching fails. Then, step 9 is executed.

[0068] Step 9: traversing the third character 'word2' of the string, obtaining its character type 1 (Chinese character), and the character type matching is successful. In the dictionary tree, the first-level node with character type 1 (Chinese character) is matched, which is represented as node1, and step 10 is entered.

[0069] Step 10: setting the detection state to the dictionary detection state, and executing step 11.

[0070] ​Step 11, dictionary tree change, i.e. in the dictionary tree, the next level node directly connected to the node node1 is located, and character matching is performed: i.e. whether the character of the node node2 matches the current character 'word2', at this time, the matching is successful, and the current character 'word2' and the position in the string is recorded as 2; Step 12 is executed;

[0071] Step 12, the detection state is set to the dictionary detection state; in the dictionary tree, the node node2 has a next level node, and the next level node thereof is located as node3, and step 13 is executed;

[0072] Step 13, the 4th character 'word3' of the string is traversed, and its character type is obtained as 1 (Chinese character), which is the same as the character type of the node node3 in the dictionary tree, therefore, the character type matching is successful, and step 14 is executed;

[0073] Step 14, the detection state is set to the dictionary detection state; in the dictionary tree, the next level node of the node node3 is located as the node node4, and the character 'word3' and the node node4 perform character matching, in this example, the character matching is successful, and the current character 'word3' and the position in the string are recorded as 3; Step 15 is executed;

[0074] Step 15, the detection state is set to the dictionary detection state; in the dictionary tree, the node node4 has a next level node, and the next level node thereof is located as node5, and step 16 is executed;

[0075] Step 16, the 5th character 'word4' of the string is traversed, and its character type is obtained as 1 (Chinese character), which is the same as the character type of the node node5 in the dictionary tree, therefore, the character type matching is successful, and step 17 is executed;

[0076] Step 17, the detection state is set to the dictionary detection state; in the dictionary tree, the next level node of the node node5 is located as the node node6, and the character 'word4' and the node node6 perform character matching, in this example, the character matching is successful, and the current character 'word4' and the position in the string are recorded as 4; Step 18 is executed;

[0077] Step 18, the 6th character 'a' of the string is traversed, and its character type is obtained as 2 (letter), which is different from the character type of the node node7 in the dictionary tree, at this time, the detection state is the dictionary detection state, therefore, the position of the located node node7 in the dictionary tree is kept unchanged, and step 19 is executed;

[0078] Step 19, traverse the 7th character 'b' of the string, and obtain its character type as 2 (letter), which is different from the character type of node node7 in the trie tree. At this time, the detection status is the dictionary detection status. Therefore, the position of the located node node7 in the trie tree remains unchanged, and step 20 is executed;

[0079] Step 20, traverse the 8th character '字5' of the string, and obtain its character type as 2 (letter), which is the same as the character type of node node7 in the trie tree. Set the detection status to the dictionary detection status, locate the next-level node of node node7, that is, node node8, and perform character matching. In this example, the character matching is successful, and record the current character '字5' and its position in the string as 7; execute step 21;

[0080] Step 21, traverse the 9th character '字6' of the string, and obtain its character type as 2 (letter), which is the same as the character type of node node9 in the trie tree. Set the detection status to the dictionary detection status, locate the next-level node of node node9, that is, node node 10 , and perform character matching. In this example, the character matching is successful, and record the current character '字6' and its position in the string as 8.

[0081] Therefore, for the to-be-tested string "字0字1字2字3字4ab字5字6", where 字0字1字2字3字4字5字6 respectively represent 7 Chinese characters, according to the trie tree, the finally recorded character sequence is: 字2字3字4字5字6, and their character positions are: 2, 3, 4, 7, 8. That is to say, the string 字2字3字4字5字6 is the string that needs to be filtered out by the trie tree positioning. Through the present invention, even if the string 字2字3字4字5字6 contains other characters such as "ab", the present invention still successfully identifies the string that needs to be filtered out, and the filtering effect is good.

[0082] ​​​​​​​​​​​​​​​

[0087] The application uses the matching mode of the dictionary tree structure, adds the character type node to the tree node, can more efficiently match the keyword, reduces the size of the dictionary volume, and uses the special character type matching rule to successfully prevent bypassing of other types of characters mixed in the middle of the keywords such as 'word2word3word4abword5word6'.

[0088] The above only describes the preferred embodiments of the present application. It should be noted that those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements should be considered within the scope of the present application.

Claims

1. A method of high performance anti-bypass keyword filtering, characterized in that, The method comprises the following steps: Step 1, defining character types; Step 2, generating a dictionary tree for all keywords to be filtered; in the dictionary tree, assuming that any keyword is composed of n characters in sequence, which are respectively: the 0th position character, the 1st position character, …, the (n-1)th position character; the representation of the keyword in the dictionary tree is formed by cascading 2n nodes in sequence, wherein the 2n nodes represent respectively: the character type of the 0th position character, the 0th position character, the character type of the 1st position character, the 1st position character, …, the character type of the (n-1)th position character, and the (n-1)th position character; Step 3, traverse the string to be detected, if the current position does not exist character, representing the end of the traversal string, the detection ends; if the current position exists character, read the current character char i , according to the character type defined in step 1, obtain the character type type i of the current character char i , execute step 4; Step 4, in the dictionary tree, judge whether a node with character type type i corresponding node, if possible, assume the matched node is node node j Step 5 is then executed; if not, step 8 is executed Step 5, setting the detection state to a dictionary detection state, and executing Step 6; Step 6, in the dictionary tree, locate the node node j of the next level directly connected node, denoted as node node j+1 Then, determine whether the character represented by node node j+1 matches the current character char i , if it matches, record the current character char i and the position of the current character char i in the string, then execute step 7; if it does not match, set the detection state to no detection state, clear the record, and reset the dictionary tree to the initial state, then return to step 3 to traverse the next character of the string; Step 7: Set the detection status to "in progress" in dictionary detection; in the trie, determine the node. j+1 Check if there is a directly connected node at the next level. If not, the trie is complete and the check ends. If there is, return to step 3 and traverse the next character of the string. Step 8, judging the current detection state; if the current detection state is the dictionary detection state, the current positioned node position in the dictionary tree remains unchanged, and Step 3 is returned to traverse the next character of the string; if the current detection state is the no detection state, the dictionary tree is reset to the initial state, and then Step 3 is returned to traverse the next character of the string; Through Steps 3 to 8, the characters and their positions in the string are finally recorded, and the recorded characters are arranged according to their positions in the string to form a character sequence, which is the detected character sequence to be filtered in the detected string.

2. The method of high performance anti-bypass keyword filtering of claim 1, wherein, The character types include four types, which are respectively: a Chinese character type, an English letter character type, a number character type, and an other character type.

3. The method of high performance anti-bypass keyword filtering of claim 2, wherein, The English letter character type includes a capital English letter character type and a small letter English letter character type.

4. A system for high performance anti-bypass keyword filtering, characterized in that, The method comprises: a character type defining module for defining character types; a dictionary tree generating module for generating a dictionary tree for all keywords to be filtered; in the dictionary tree, assuming that any keyword is composed of n characters in sequence, which are respectively: the 0th position character, the 1st position character, …, the (n-1)th position character; the representation of the keyword in the dictionary tree is formed by cascading 2n nodes in sequence, wherein the 2n nodes represent respectively: the character type of the 0th position character, the 0th position character, the character type of the 1st position character, the 1st position character, …, the character type of the (n-1)th position character, and the (n-1)th position character; a bypass prevention keyword detection module for identifying keywords from a string to be detected by using the following method: Step 3, traverse the string to be detected, if the current position does not exist a character, it represents the end of the string and the detection is over; if the current position exists a character, read the current character char i , according to the character type defined in step 1, obtain the character type type i of the current character char i , execute step 4; Step 4, in the dictionary tree, judge whether a node matching to the character type type i corresponding node, if possible, assume the matched node is node node j Step 5 is then executed; if not, step 8 is executed Step 5, setting the detection state to a dictionary detection state, and executing Step 6; Step 6, in the dictionary tree, locate the node node j of the next level directly connected node, denoted as node node j+1 Then, determine whether the character represented by node node j+1 matches the current character char i , if it matches, record the current character char i and the position of the current character char i in the string, then execute step 7; if it does not match, set the detection state to no detection state, clear the record, and reset the dictionary tree to the initial state, then return to step 3 to traverse the next character of the string; Step 7: Set the detection status to "in progress" in dictionary detection; in the trie, determine the node. j+1 Check if there is a directly connected node at the next level. If not, the trie is complete and the check ends. If there is, return to step 3 and traverse the next character of the string. Step 8, judging the current detection state; if the current detection state is the dictionary detection state, the current positioned node position in the dictionary tree remains unchanged, and Step 3 is returned to traverse the next character of the string; if the current detection state is the no detection state, the dictionary tree is reset to the initial state, and then Step 3 is returned to traverse the next character of the string; Through Steps 3 to 8, the characters and their positions in the string are finally recorded, and the recorded characters are arranged according to their positions in the string to form a character sequence, which is the detected character sequence to be filtered in the detected string.

Images