Sensitive word recognition method and sensitive word tree construction method

Abstract

The application discloses a sensitive word recognition method and a sensitive word tree construction method. The recognition method comprises the following steps: in step S11, a character in a text to be detected is read as a first character, and a root node of a sensitive word tree is taken as an initial parent node; in step S12, a character package value of the first character is obtained from a dictionary table, and a child node data table of the parent node is traversed in the sensitive word tree to determine a child node corresponding to the character package value of the first character; in step S13, it is judged whether the child node is a leaf node, if yes, a sensitive word is output, and step S15 is executed; if not, step S14 is executed; in step S14, a next character in the text to be detected is taken as a new first character, and the child node is taken as a new parent node, and step S12 is executed; in step S15, it is judged whether the text to be detected is completely detected, if yes, the method is ended, and if not, the method returns to step S11. The application provides a lightweight solution for quickly recognizing sensitive words.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, specifically to a method for identifying sensitive words and a method for constructing a sensitive word tree, a computing device, and a computer storage medium. Background Technology

[0002] In today's internet environment, text content is a crucial carrier of information. Forums, websites, and applications allow users to independently edit and upload large amounts of text content. Administrators of these platforms have a responsibility to filter this text content to ensure a healthy and harmonious online environment. Sensitive word identification and filtering is a highly effective method.

[0003] One common method for filtering sensitive words is to maintain a sensitive word database to find sensitive words in text. While this algorithm is simple, it requires maintaining a large database, which can expand rapidly, especially when dealing with various variations of sensitive words. Another method is to use regular expressions and Natural Language Processing (NLP), but this also has problems such as the inability to effectively identify variations of sensitive words.

[0004] In summary, existing sensitive word filtering schemes have the following problems: they cannot effectively handle the transformation of sensitive words, such as character splitting, mixed Chinese pinyin, homophones, and homographs; when using a sensitive word database to handle sensitive word transformations, the database expands by tens or hundreds of times, leading to a sharp increase in memory consumption and a sharp decrease in data construction performance; when dealing with sensitive word transformations, NLP algorithms experience a sharp increase in training samples, even to an unacceptable degree; in addition, existing schemes cannot effectively handle the case where sensitive words are not adjacent. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a method for identifying sensitive words and a method for constructing a sensitive word tree that overcomes or at least partially solves the above problems.

[0006] According to one aspect of the present invention, a method for identifying sensitive words is provided, comprising:

[0007] Step S11: Read the characters in the text to be detected in sequence as the first character, and use the root node of the sensitive word tree as the initial parent node;

[0008] Step S12: Obtain the character encapsulation value of the first character from the dictionary table, traverse the child node data table of the parent node in the sensitive word tree, and determine the child node corresponding to the character encapsulation value of the first character.

[0009] Step S13: Determine whether the child node is a leaf node. If yes, output the sensitive word and proceed to step S15; otherwise, proceed to step S14.

[0010] Step S14: Take the next character in the text to be detected as the new first character, and take the child node as the new parent node, and execute step S12.

[0011] Step S15: Determine whether all the text to be detected has been detected. If yes, the method ends; otherwise, return to step S11.

[0012] Optionally, in step S12, if no child node corresponding to the character encapsulation value of the first character is found, the method further includes:

[0013] Extract at least one variant character information of the first character from the character encapsulation value of the first character;

[0014] Based on the at least one deformed character information, obtain the character encapsulation value of at least one deformed character from the dictionary table;

[0015] Traverse the child node data table of the parent node in the sensitive word tree to find the child node corresponding to the character encapsulation value of any one of the at least one deformed characters;

[0016] The child node corresponding to the character encapsulation value of any of the deformed characters is taken as the child node corresponding to the character encapsulation value of the first character.

[0017] Optionally, before step S12, the method further includes:

[0018] Determine whether the first character is an English character;

[0019] If so, continue reading one or more English characters after the first character to form an English character group. If the English character group is Chinese Pinyin, then traverse the child node data table of the parent node in the sensitive word tree to find the child node corresponding to the character encapsulation value containing the Chinese Pinyin.

[0020] Optionally, before step S14, the method further includes:

[0021] Step S21: Determine whether the child node contains connected child nodes; if yes, proceed to step S22; if no, proceed to step S14.

[0022] Step S22: Starting from the position of the first character in the text to be detected, read the second character sequentially. Using the connection node corresponding to the connection child node as the new parent node, query the character encapsulation value of the second character from the dictionary table. Traverse the child node data table of the parent node in the sensitive word tree to find the child node corresponding to the character encapsulation value of the second character. If no child node corresponding to the character encapsulation value of the second character is found, the next character is used as the new second character, and step S22 continues. If a child node corresponding to the character encapsulation value of the second character is found, proceed to step S13.

[0023] Optionally, before proceeding to step S22 with the next character as the new second character, the method further includes:

[0024] Determine whether the termination condition for non-continuous sensitive word matching is met. If so, proceed to step S11.

[0025] The following character is used as the new second character, and the execution of step S22 continues. Further, if it is determined that the termination condition of non-continuous sensitive word matching is not met, the following character is used as the new second character, and the execution of step S22 continues.

[0026] Optionally, before step S11, the method further includes:

[0027] The text to be detected is preprocessed, including but not limited to: converting non-text characters into fixed placeholders, converting between traditional and simplified Chinese characters, and converting emoticons into text characters.

[0028] According to another aspect of the present invention, a method for constructing a sensitive word tree is provided, comprising:

[0029] Step S31: Create the root node of the sensitive word, use the root node as the initial parent node, and read the first character of the sensitive word as the initial third character;

[0030] Step S32: Obtain the character encapsulation value of the third character and the character encapsulation value of at least one associated character of the third character from the dictionary table, create multiple child nodes of the parent node, and write the character encapsulation value of the third character and the character encapsulation value of at least one associated character of the third character into the child node data table of the parent node.

[0031] Step S33: Determine whether the third character is the last character of the sensitive word. If yes, set the multiple child nodes as leaf nodes; if no, set each child node as the new parent node and the next character of the sensitive word as the new third character, and execute step S32.

[0032] Optionally, before step S32, the method further includes:

[0033] Determine whether the third character is a sensitive word connector;

[0034] If so, create the connecting child node of the parent node, and create the connecting node corresponding to the sensitive word connector, establish the parent-child relationship between the connecting node and the connecting child node, take the connecting node as the new parent node, take the next character as the new third character, and execute step S32;

[0035] If not, proceed directly to step S32.

[0036] According to another aspect of the present invention, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0037] The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the sensitive word identification method according to any of the preceding claims, and / or, the executable instruction causes the processor to perform an operation corresponding to the sensitive word tree construction method according to any of the preceding claims.

[0038] According to another aspect of the present invention, a computer storage medium is provided, the storage medium storing at least one executable instruction, the executable instruction causing a processor to perform an operation corresponding to the sensitive word identification method according to any one of the preceding claims, and / or, the executable instruction causing the processor to perform an operation corresponding to the sensitive word tree construction method according to any one of the preceding claims.

[0039] According to the sensitive word identification method of the present invention, the identification method of sensitive words is flexibly expanded by designing a Chinese character dictionary and a sensitive word tree. It can quickly identify sensitive words without causing a sharp increase in memory, thus improving memory utilization efficiency. It is especially suitable for content review scenarios.

[0040] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0042] Figure 1 A schematic diagram of the structure of a dictionary table provided in an embodiment of the present invention is shown;

[0043] Figure 2 A schematic diagram of the sensitive word tree structure provided in an embodiment of the present invention is shown;

[0044] Figure 3 A flowchart illustrating a method for constructing a sensitive word tree according to an embodiment of the present invention is shown;

[0045] Figure 4 A flowchart illustrating a method for constructing a sensitive word tree according to another embodiment of the present invention is shown;

[0046] Figure 5 A flowchart of a sensitive word identification method provided by an embodiment of the present invention is shown;

[0047] Figure 6 A flowchart of a sensitive word identification method provided by another embodiment of the present invention is shown;

[0048] Figure 7 A flowchart of a method for searching non-continuous sensitive words according to an embodiment of the present invention is shown;

[0049] Figure 8 This diagram illustrates the structure of a sensitive word tree construction apparatus according to an embodiment of the present invention.

[0050] Figure 9 A schematic diagram of the structure of a sensitive word recognition device provided in an embodiment of the present invention is shown;

[0051] Figure 10 A schematic diagram of the structure of a computing device provided in an embodiment of the present invention is shown. Detailed Implementation

[0052] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0053] This invention provides an embodiment of a method for constructing a sensitive word tree, which is applied in a computing device. The computing device includes, but is not limited to, servers, personal computers, laptops, and various smart terminal devices. This computing device can perform at least one of the following functions: sensitive word management, dictionary design, sensitive word tree construction, and sensitive word recognition in text content.

[0054] In an embodiment of the present invention, in order to construct a sensitive word tree, the sensitive words or phrases to be identified are first sorted out to determine the sensitive words to be identified.

[0055] Optionally, to facilitate subsequent work, this embodiment of the invention divides sensitive words into two categories: continuous sensitive words and non-continuous sensitive words. Continuous sensitive words can only be matched when all characters of the sensitive word appear consecutively in the target text. Non-continuous sensitive words are based on continuous sensitive words but add a connector (such as the symbol "||"). The characters on both sides of the connector in the sensitive word are not adjacent in the target text (such as being separated by several Chinese characters) and can still be matched. The above two types of sensitive words can each include Chinese character variations, such as Pinyin, initial letters, left-right split characters, and variant characters (including but not limited to homophones, variant characters, and homonyms).

[0056] To facilitate the expression and storage of the aforementioned variant characters, a dictionary table is also designed, which can be an in-memory hash table. Within this dictionary table, Chinese characters are used as keys, and each Chinese character, along with its associated Pinyin, left-right split characters, variant characters, and other information, is encapsulated together as a character encapsulation value. For example... Figure 1 As shown, in the dictionary table, Chinese characters are used as the table keys, and the table values ​​correspond to the encapsulated character values.

[0057] It is important to emphasize that the set of variant characters in the character encapsulation value stores the address of the corresponding variant character's value in the dictionary table. Through this dictionary structure design, the character encapsulation value corresponding to the character can be quickly obtained through the character, thereby obtaining information such as Chinese Pinyin, left and right split characters, and variant characters from the character encapsulation value, which is beneficial for simplifying the subsequent process and reducing memory consumption.

[0058] In addition to constructing a dictionary table, a sensitive word tree can also be constructed. Preferably, in this embodiment, the sensitive word is first split into individual Chinese characters or alphabetic characters, with each character corresponding to a node in the sensitive word tree. The sensitive word is then parsed into a tree structure composed of these nodes. For example... Figure 2 The diagram shows the structure of a sensitive word tree, which includes four types of nodes: root node, child node, connecting node, and leaf node.

[0059] The root node is the starting point for sensitive word construction and recognition. The first character parsed from a sensitive word is encapsulated into a child node, and its encapsulation value in the dictionary is used as the key in the child node data table of the root node. This child node data table is preferably a hash table. Furthermore, the keys in the child node data table store not only the character's own encapsulation value but also the encapsulation values ​​of at least one associated character. This at least one associated character includes, but is not limited to, the first letter of the pinyin and / or split characters. The value corresponding to each key in the child node data table is the child node address; by querying the key, the corresponding child node can be indexed. In addition, the root node may also store a set of pinyin for the child nodes.

[0060] like Figure 2 As shown, the child nodes store information such as node values, a child node data table, and connected child nodes. The node value stores the address of the character's encapsulation value in the dictionary table for the character corresponding to the current child node. This address allows for quick retrieval of associated character values, Chinese pinyin, split characters, and variant characters (variable characters are optional), thus saving memory space. The child node data table stores the next-level nodes of the current child node. It should be noted that the child nodes stored in this data table are used for searching for consecutive sensitive words. These child nodes can also include connected child nodes, which store the connected nodes under that child node for searching for non-consecutive characters.

[0061] A connection node is a special type of node that does not have node values ​​or other information, but only a child node data table. The child nodes stored in this child node data table are used for searching for non-continuous characters in sensitive words.

[0062] A leaf node is a node constructed from the last character of a sensitive word; a leaf node is also a special type of child node.

[0063] It should be noted that the dictionary table and sensitive word tree mentioned above can be updated in real time as needed.

[0064] For sensitive words of different types and lengths, the method for constructing a sensitive word tree based on information from the dictionary table can be as follows: Figure 3 Perform the steps shown:

[0065] Step S31: Create the root node of the sensitive word, use the root node as the initial parent node, and read the first character of the sensitive word as the initial third character;

[0066] Step S32: Obtain the character encapsulation value of the third character and the character encapsulation values of at least one associated character of the third character from the dictionary table, create multiple child nodes of the parent node, and write the character encapsulation value of the third character and the character encapsulation values of at least one associated character of the third character into the child node data table of the parent node;

[0067] Step S33: Determine whether the third character is the last character of the sensitive word. If so, set the multiple child nodes as leaf nodes; if not, respectively use each child node as a new parent node, use the next character of the sensitive word as the new third character, and execute Step S32.

[0068] It should be noted that in the embodiments of the present invention, the first character, the second character (mentioned below), and the third character are used as the currently processed characters. The expressions of these three characters are only for distinguishing the currently processed characters in each embodiment.

[0069] To construct a sensitive word tree, a pre-constructed dictionary table is required. Optionally, the dictionary table can be pre-stored in memory to improve query efficiency.

[0070] In Step S31, create the root node of the sensitive word. This root node does not correspond to any character of the sensitive word and represents the starting node of a certain sensitive word.

[0071] In Step S32, after creating the root node of a certain sensitive word first, use the root node as the parent node and create the child nodes of the root node according to the first character read.

[0072] Taking the sensitive word "usury" as an example, the first character is "高". In actual execution, the character "高" can be read first, create a child node according to the read character "高", and write the character encapsulation value of the character "高" as a key into the child node data table of the root node; at the same time, create other child nodes of the root node according to at least one associated character of the first character. The at least one associated character includes but is not limited to: the first letter of the pinyin and / or the split character. The split character can be a left-right split character or an up-down split character. For example, create a child node according to the first letter "G" of the pinyin of the character "高", and write the character encapsulation value of the character "G" as a key into the child node data table of the root node.

[0073] For the case where the sensitive word includes multiple characters, Step S32 will be iteratively executed. During the iteration, using the parent node as a variable, it is necessary to use each of the child nodes corresponding to the previous character in the sensitive word as the parent node to create the child nodes corresponding to the current character.

[0074] Still taking "usury" as an example, in the second iteration process, taking the corresponding child nodes such as "high" and "G" as the parent nodes respectively, multiple child nodes of the parent node are created, that is, the child nodes corresponding to "interest" and the initials of the pinyin or the split characters related to "interest" are created, and the character encapsulated values of "interest" and the initials of the pinyin or the split characters related to "interest" (such as "grain" and "knife") are written into the child node data tables of the corresponding child nodes respectively. In the third iteration process, taking the child nodes corresponding to "interest" and the initials of the pinyin or the split characters related to "interest" as the parent nodes respectively, multiple child nodes of the parent node are created, that is, the child nodes corresponding to "loan" and the initials of the pinyin or the split characters related to "loan" (such as "substitute" and "shell") are created, and the character encapsulated values of "loan" and the initials of the pinyin or the split characters related to "loan" are written into the child node data tables of the child nodes corresponding to "interest" and the initials of the pinyin or the split characters related to "interest" respectively.

[0075] In step S33, it is judged whether the third character is the last character of the sensitive word. For example, "loan" is the last character of "usury", then the child nodes corresponding to "loan" and the initials of the pinyin or the split characters related to "loan" are set as leaf nodes, thus completely creating the sensitive word tree of the sensitive word "usury".

[0076] It should be noted that at least one associated character of the third character in the embodiment of the present invention may not include variant characters, but place the addresses of the variant characters in the character encapsulated values of the corresponding characters. When recognition is required, it is necessary to rely on the dictionary table to extract the variant character set from the dictionary table, thus avoiding the excessive expansion of the sensitive word tree and affecting the query efficiency of the sensitive word tree.

[0077] However, it should be understood that at least one associated character of the third character in the embodiment of the present invention may also include variant characters, that is, in addition to creating the child nodes corresponding to the initials of the pinyin and / or the split characters of the current character in the sensitive word tree, the child nodes corresponding to the variant characters of the current character can also be created together. In this way, when recognition is performed, there is no need to extract the variant character set from the dictionary table, and the recognition of variant characters can be achieved relying on the sensitive word tree.

[0078] The method for creating a continuous sensitive word tree is specifically described in detail in the above embodiments. Further, in another embodiment, a discontinuous sensitive word tree can also be created.

[0079] To construct a non-contiguous sensitive word tree, connection nodes need to be set, specifically as follows: After step S31 and before step S32, the process further includes: First, determining whether the third character is a sensitive word connector; if yes, if the parent node's connection child node is empty, then creating the parent node's connection child node and creating the connection node corresponding to the sensitive word connector, establishing a parent-child relationship between the connection node and the connection child node, using the connection node as the new parent node, and using the next character as the new third character, and then executing step S32; if no, then directly executing step S32.

[0080] In one optional embodiment, a method as described is disclosed. Figure 4 The complete process of constructing a sensitive word tree shown includes the creation process of the root node, child nodes, leaf nodes, and connection nodes, and can be universally applied to the construction of various types of sensitive word trees. First, a root node needs to be created. In step 1, the first character of the sensitive word is read, and this root node is used as the initial parent node. In step 2, the obtained character (i.e., the third character) is used to retrieve the character encapsulation value from the memory dictionary table, and this encapsulation value is used as the node value to create child nodes. Additionally, child nodes are created using the first letter of the pinyin or split characters in the encapsulation value. In steps 3, 4, and 5, all child nodes of the parent node are traversed, and each child node is used as the current child node for subsequent steps. In step 6, it is determined whether the current character is the last character of the sensitive word. In step 7, the current child node is set as a leaf node, ending the creation of the sensitive word tree. In step 8, the next character of the sensitive word is retrieved, and the current child node is set as the parent node for subsequent operations. In step 9, it is determined whether the current character is a sensitive word connector. If not, step 2 is skipped; if so, step 10 is executed, using the connector child node as the parent node to create a connector node, which is then used as the new parent node. In step 11, the next character of the sensitive word is retrieved, and step 2 is skipped.

[0081] As Figure 4 The steps disclosed in the document, the process of constructing the sensitive word tree, involves placing each character of the sensitive word into a corresponding position according to the steps described above. Figure 2 The process of creating a sensitive word tree is shown.

[0082] Figure 5 A flowchart of an embodiment of the sensitive word identification method of the present invention is shown. The method uses the sensitive word tree in the above embodiment, and the identification process and the creation process have a corresponding relationship. The method is also applied to computing devices.

[0083] like Figure 5 As shown, the identification method includes the following steps:

[0084] Step S11: Read the characters in the text to be detected one by one as the first character, and use the root node of the sensitive word tree as the initial parent node.

[0085] After obtaining the text to be detected, each character can be read one by one as the first character until the entire text to be detected is read. And, query the first character in the root nodes of each pre-created sensitive word tree through the information of the character encapsulation value. If it does not exist, use the next character in the text to be detected as the first character and query again. If it exists, use the corresponding root node of the sensitive word tree as the initial parent node, and determine whether there is a string in the text to be detected that is the same as the sensitive word in the pre-created sensitive word tree through the following steps.

[0086] For example, take the sentence "Provide usury services" in the text to be detected as an example, where "usury" is a sensitive word, and the corresponding sensitive word tree is constructed. During the recognition process, the characters should be read one by one. Since no corresponding sensitive word is found in the sensitive word tree according to "Provide" and "usury", until the character "usury" is read, it starts to match the sensitive word tree "usury".

[0087] Step S12: Obtain the character encapsulation value of the first character from the dictionary table, traverse the child node data table of the parent node in the sensitive word tree, and determine the child node corresponding to the character encapsulation value of the first character.

[0088] Query the character encapsulation value of "usury" from the dictionary table, traverse the root nodes of each sensitive word tree, and check which root node's child node data table stores the character encapsulation value of "usury". After finding it, the sensitive word tree is located. Optionally, there may be multiple sensitive word trees corresponding to the first character. In the subsequent processing flow, it is necessary to perform recognition and matching for each sensitive word tree respectively.

[0089] Step S13: Determine whether the child node is a leaf node. If so, output the sensitive word and execute Step S15. If not, execute Step S14.

[0090] In the above example, when the first character is "usury", the current child node is not a leaf node, and at this time, Step S14 needs to be executed. After iterating multiple times, when the first character is "services", the current child node is a leaf node, output the sensitive word "usury", and execute Step S15. If the current child node is not a leaf node, continue to detect the sensitive word according to Step S14.

[0091] Step S14: Use the next character in the text to be detected as the new first character, use the child node as the new parent node, and execute Step S12.

[0092] According to this step, still taking "usury" as an example, use the next character "i" of "gāo" as the new first character, and use the child node corresponding to "gāo" as the new parent node, and iteratively execute step S12.

[0093] Step S15, determine whether all of the text to be detected has been detected. If so, the method ends; if not, return to step S11.

[0094] After outputting a complete sensitive word, it is necessary to determine whether the detection of the text to be detected is completed. If the detection is completed, the execution of this method ends. Otherwise, taking the sentence "Provide usury services" as an example, continue to execute step S11, read the next character "fú", and continue the recognition process.

[0095] Furthermore, in step S12, if no child node corresponding to the character encapsulation value of the first character is found, the method further includes a step of judging deformed characters:

[0096] Extract at least one deformed character information of the first character from the character encapsulation value of the first character; according to the at least one deformed character information, obtain the character encapsulation values of at least one deformed character from the dictionary table; traverse the child node data table of the parent node in the sensitive word tree, and find the child node corresponding to the character encapsulation value of any deformed character; then, use the child node corresponding to the character encapsulation value of any deformed character as the child node corresponding to the character encapsulation value of the first character.

[0097] Taking "gāo lì dài" in the text to be detected as an example, that is, the deformed word of the sensitive word in the text to be detected. At this time, when iterating to the first character "lì", since "lì" does not appear in the child nodes of the corresponding sensitive word tree, this method needs to find its corresponding deformed characters according to the character encapsulation value of "lì". By traversing whether there is a certain deformed character in the deformed character set of the character encapsulation value of "lì" and the character encapsulation value of this deformed character corresponds to a child node, if it exists, it is considered that the match is successful.

[0098] In another embodiment, before step S12, after reading the first character from the text to be detected, the method further includes:

[0099] Judge whether the first character is an English character; if so, continue to read one or more English characters after the first character to form an English character group. If the English character group is Chinese pinyin, traverse each child node corresponding to the child node data table of this parent node in the sensitive word tree, and find the child node corresponding to the character encapsulation value containing the Chinese pinyin.

[0100] In this step, if the currently matched character is an English character, the subsequent character is used to merge with the current character to check if it is Chinese Pinyin. If it is Chinese Pinyin, the character encapsulation value stored in the child node is used to further determine whether there is a child node corresponding to the Chinese Pinyin.

[0101] In one or more implementations, combined with Figure 7 The non-continuous search process shown, prior to step S14, further includes:

[0102] Step S21: Determine whether the child node contains connected child nodes. For example, this can be done by detecting... Figure 2 Determine whether the child nodes in the process include connected child nodes; if yes, proceed to step S22; if no, proceed to step S14.

[0103] Specifically, in step S22, starting from the position of the first character in the text to be detected, the second character is read sequentially. The connection node corresponding to the connection child node is used as the new parent node. The character encapsulation value of the second character is queried from the dictionary table. The child node data table of the parent node is traversed in the sensitive word tree to find the child node corresponding to the character encapsulation value of the second character. If no child node corresponding to the character encapsulation value of the second character is found, the next character is used as the new second character, and step S22 is continued. If a child node corresponding to the character encapsulation value of the second character is found, step S13 is executed.

[0104] Furthermore, before proceeding to step S22 with the next character as the new second character, the method further includes: determining whether the end condition for non-continuous sensitive word matching is met, wherein the end condition includes: the maximum non-continuous detection length specified by the system or the end of the text, etc.; if so, then proceed to step S11.

[0105] The step S22 is further defined as follows: if it is determined that the end condition for non-continuous sensitive word matching is not met, then the next character is used as the new second character, and the next step S22 is executed.

[0106] In summary, for the identification of non-contiguous sensitive words, the position of the character to which the child node belongs in the text to be detected can be used as the starting point, and subsequent characters can be used as the starting positions for searching for non-contiguous sensitive words.

[0107] In addition, before performing sensitive word identification, the text to be detected can be preprocessed. Preprocessing includes, but is not limited to, converting non-text characters into fixed placeholders, converting between simplified and traditional Chinese characters, and converting emojis into text characters. After preprocessing, the preprocessed text to be detected is passed to a sensitive word tree for sensitive word identification, and the sensitive words are output. It should be noted that the sensitive word identification process in this embodiment is closely related to the construction process of the sensitive word tree.

[0108] In an optional embodiment, such as Figure 6 The diagram illustrates the overall steps of a method for identifying sensitive words. Figure 6 The embodiments can be considered as... Figure 5 Further explanation of the illustrated embodiment.

[0109] Specifically, in Figure 6 In step 2, if the currently matched character is an English character, the subsequent character is used to merge with the current character to search for whether it is Chinese Pinyin. If Chinese Pinyin can be obtained, step 4 is executed.

[0110] In step 3, the character encapsulation value corresponding to the character obtained from the target text is obtained from the memory dictionary table. If the corresponding child node can be found directly from the child node hash table of the parent node through the character encapsulation value, the subsequent steps are executed. If the child node is not found directly through the character encapsulation value, the variant character stored in the character encapsulation value is used to search for the child node from the child node hash table of the parent node and the subsequent steps are executed in sequence.

[0111] In step 4, the Chinese pinyin is matched against each child node in the hash of the child nodes contained in the parent node. If the Chinese pinyin of the character encapsulation value stored in the child node matches the current pinyin, the subsequent steps are executed.

[0112] In step 5, it is determined whether a child node has been found. If not, the current search ends and the branch search described in the other steps continues.

[0113] In step 6, determine whether the found child node is a leaf node. If it is a leaf node, output the matched sensitive word; otherwise, proceed to step 7.

[0114] In step 7, it is determined whether the target text has been checked. If it is not found, the current search ends and the branch search described in other steps continues.

[0115] If, in step 8, a child node is found to contain a connecting child node, it indicates that there is a connecting node under that child node. In this case, a non-contiguous sensitive word matching operation should be performed. For details on this operation, please refer to [link to relevant documentation]. Figure 7 .

[0116] like Figure 7The diagram shows the steps for performing a non-contiguous search for sensitive words. Figure 6 In step 8, if a child node has connected child nodes, then proceed to... Figure 7 The processing branch shown uses the position of the character to which the child node belongs in the text to be detected as the starting point, and uses subsequent characters as the starting positions to continuously search for sensitive words. Figure 7 The third step, the sensitive word search, refers to... Figure 6 The method for finding sensitive words in [the context of the text].

[0117] In summary, compared with the prior art, the technical solution disclosed in this invention has the following characteristics and advantages:

[0118] A Chinese character dictionary table was constructed to maintain Chinese character information. The dictionary table makes it easy to maintain and query information such as the pinyin, character splitting, and variant characters of Chinese characters.

[0119] The sensitive word tree is constructed by automatically building sensitive word pinyin nodes and splitting byte points in addition to the character nodes themselves, which effectively improves the recognition efficiency of sensitive words.

[0120] The sensitive word tree can indirectly support the search of variant characters in its construction nodes, without needing to create a node for each variant character, thus reducing the number of nodes in the tree and saving memory.

[0121] Sensitive word recognition supports the search for non-contiguous sensitive words, which can effectively deal with situations where attackers split sensitive words into multiple parts.

[0122] The system supports non-continuous recognition of sensitive words, thus addressing situations where a single word is not a sensitive word, but multiple words combined in a sentence become sensitive words, thereby improving the accuracy of sensitive word recognition.

[0123] The sensitive word recognition supports searching by Chinese Pinyin and by the first letter of Pinyin, which can effectively deal with unauthorized users' Pinyin and first letter distortion of sensitive words.

[0124] The sensitive word recognition feature supports searching for modified versions of sensitive words, effectively addressing the identification of modified versions of sensitive words by unauthorized users, such as homographs, variant characters, and homophones.

[0125] The dictionary tables are loaded into memory as hash dictionary tables, and these dictionary table structures are reused when building the sensitive word tree to save memory.

[0126] Sensitive word recognition adds a preprocessing method to improve the accuracy of sensitive word recognition by preprocessing the text to be detected.

[0127] like Figure 8 As shown, this embodiment of the invention also provides a sensitive word construction device 800, wherein the construction device 800 includes:

[0128] The root node creation module 810 is adapted to create the root node of the sensitive word, using the root node as the initial parent node, and reading the first character of the sensitive word as the initial third character;

[0129] The child node creation module 820 is adapted to obtain the character encapsulation value of the third character and the character encapsulation value of at least one associated character of the third character from a dictionary table, create multiple child nodes of the parent node, and write the character encapsulation value of the third character and the character encapsulation value of at least one associated character of the third character into the child node data table of the parent node.

[0130] The iterative creation module 830 is adapted to determine whether the third character is the last character of the sensitive word. If so, the multiple child nodes are set as leaf nodes; if not, each child node is set as a new parent node, and the next character of the sensitive word is set as the new third character, triggering the child node creation module 820.

[0131] In one embodiment, the apparatus further includes a connection node creation module, which is adapted to:

[0132] Determine whether the third character is a sensitive word connector;

[0133] If yes, create the connecting child node of the parent node and the connecting node corresponding to the sensitive word connector, establish the parent-child relationship between the connecting node and the connecting child node, take the connecting node as the new parent node, take the next character as the new third character, and trigger the child node creation module 820; if no, directly trigger the child node creation module 820.

[0134] like Figure 9 As shown, this embodiment of the invention provides a sensitive word identification device 900, which includes:

[0135] The detection start module 910 is adapted to read the characters in the text to be detected sequentially as the first character, and use the root node of the sensitive word tree as the initial parent node;

[0136] The traversal recognition module 920 is adapted to obtain the character encapsulation value of the first character from the dictionary table, traverse the child node data table of the parent node in the sensitive word tree, and determine the child node corresponding to the character encapsulation value of the first character.

[0137] The sensitive word output module 930 is adapted to determine whether the child node is a leaf node. If it is, the sensitive word is output, triggering the end judgment module 950; otherwise, the iterative recognition module 940 is triggered.

[0138] The iterative recognition module 940 is adapted to use the next character in the text to be detected as the new first character and the child node as the new parent node to trigger the traversal recognition module 920.

[0139] The termination judgment module 950 is adapted to determine whether all the text to be detected has been detected. If so, the recognition ends; otherwise, the detection start module 910 is triggered again.

[0140] In one embodiment, the device further includes a deformed character recognition module, adapted to:

[0141] Extract at least one variant character information of the first character from the character encapsulation value of the first character;

[0142] Based on the at least one deformed character information, obtain the character encapsulation value of at least one deformed character from the dictionary table;

[0143] Traverse the child node data table of the parent node in the sensitive word tree to find the child node corresponding to the character encapsulation value of any deformed character;

[0144] The child node corresponding to the character encapsulation value of any deformed character is taken as the child node corresponding to the character encapsulation value of the first character.

[0145] In one embodiment, the device further includes a pinyin recognition module, adapted to:

[0146] Determine whether the first character is an English character;

[0147] If so, continue reading one or more English characters after the first character to form an English character group. If the English character group is Chinese Pinyin, then traverse the child node data table of the parent node in the sensitive word tree to find the child node corresponding to the character encapsulation value containing the Chinese Pinyin.

[0148] In one embodiment, the device further includes a non-continuous word recognition module, adapted to:

[0149] The judgment submodule is adapted to determine whether the child node contains a connected child node; if yes, the iteration submodule is triggered; if no, the iteration identification module 940 is triggered.

[0150] The iterative submodule is adapted to read the second character sequentially starting from the position of the first character in the text to be detected, taking the connection node corresponding to the connection child node as the new parent node, querying the character encapsulation value of the second character from the dictionary table, traversing the child node data table of the parent node in the sensitive word tree, searching for the child node corresponding to the character encapsulation value of the second character, if no child node corresponding to the character encapsulation value of the second character is found, then the next character is taken as the new second character, and this module is triggered again; if a child node corresponding to the character encapsulation value of the second character is found, the sensitive word output module 930 is triggered.

[0151] Optionally, the non-continuous word recognition module is also suitable for:

[0152] Determine whether the termination condition for non-continuous sensitive word matching is met. If so, trigger the detection start module 910.

[0153] If the following character is used as the new second character, and it is determined that the end condition for non-continuous sensitive word matching is not met, then the following character is used as the new second character, triggering the iterative submodule.

[0154] In one embodiment, the apparatus further includes a preprocessing module adapted to:

[0155] The text to be detected is preprocessed, including but not limited to: converting non-text characters into fixed placeholders, converting between traditional and simplified Chinese characters, and converting emoticons into text characters.

[0156] In summary, the device disclosed in this invention can quickly construct a sensitive word tree and identify sensitive words; it supports the transformation of sensitive words such as variant words and pinyin, without significantly increasing the number of nodes in the sensitive word tree or causing a sharp increase in memory; it supports non-continuous searching of sensitive words, improving the accuracy of sensitive word identification; the preprocessing before sensitive word identification improves the accuracy of sensitive word identification by preprocessing the text; and the dictionary table flexibly expands the construction and identification of sensitive words, reducing memory usage.

[0157] This invention provides a non-volatile computer storage medium storing at least one executable instruction that can execute the sensitive word tree construction method or the sensitive word identification method in any of the above method embodiments.

[0158] Figure 10 The diagram shows a structural schematic of an embodiment of the computing device of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

[0159] like Figure 10As shown, the computing device may include: a processor 1002, a communications interface 1004, a memory 1006, and a communications bus 1008.

[0160] The processor 1002, communication interface 1004, and memory 1006 communicate with each other via communication bus 1008. Communication interface 1004 is used to communicate with other network elements such as clients or other servers. The processor 1002 executes program 1010, specifically performing the relevant steps in the above-described method embodiments for sensitive word recognition or sensitive word tree construction for computing devices.

[0161] Specifically, program 1010 may include program code that includes computer operation instructions.

[0162] The processor 1002 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0163] Memory 1006 is used to store program 1010. Memory 1006 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0164] Specifically, program 1010 can be used to cause processor 1002 to perform operations corresponding to the embodiments of the above-described sensitive word tree construction method or sensitive word recognition method.

[0165] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of the present invention are not directed to any particular programming language. It should be understood that the content of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0166] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0167] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the embodiments of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into this detailed description, wherein each claim itself is a separate embodiment of the invention.

[0168] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0169] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0170] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0171] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for identifying sensitive words, characterized in that, include: Step S11: Read the characters in the text to be detected in sequence as the first character, and use the root node of the sensitive word tree as the initial parent node; Step S12: Obtain the character encapsulation value of the first character from the dictionary table, traverse the child node data table of the parent node in the sensitive word tree, and determine the child node corresponding to the character encapsulation value of the first character; wherein, in the dictionary table, the character is used as the key value, and the character and its associated pinyin, left and right split characters, and variant character sets are encapsulated together as the character encapsulation value; the variant character set in the character encapsulation value stores the address of the corresponding variant character value in the dictionary table; Step S13: Determine whether the child node is a leaf node. If yes, output the sensitive word and proceed to step S15; otherwise, proceed to step S14. Step S14: Take the next character in the text to be detected as the new first character, and take the child node as the new parent node, and execute step S12. Step S15: Determine whether all the text to be detected has been detected. If yes, the method ends; otherwise, return to step S11. In step S12, if no child node corresponding to the character encapsulation value of the first character is found, the method further includes: Extract at least one variant character information of the first character from the character encapsulation value of the first character; Based on the at least one deformed character information, obtain the character encapsulation value of at least one deformed character from the dictionary table; Traverse the child node data table of the parent node in the sensitive word tree to find the child node corresponding to the character encapsulation value of any one of the at least one deformed characters; The child node corresponding to the character encapsulation value of any of the deformed characters is taken as the child node corresponding to the character encapsulation value of the first character.

2. The method according to claim 1, characterized in that, Prior to step S12, the method further includes: Determine whether the first character is an English character; If so, continue reading one or more English characters after the first character to form an English character group. If the English character group is Chinese Pinyin, then traverse the child node data table of the parent node in the sensitive word tree to find the child node corresponding to the character encapsulation value containing the Chinese Pinyin.

3. The method according to any one of claims 1-2, characterized in that, Prior to step S14, the method further includes: Step S21: Determine whether the child node contains connected child nodes; if yes, proceed to step S22; if no, proceed to step S14. Step S22: Starting from the position of the first character in the text to be detected, read the second character sequentially. Using the connection node corresponding to the connection child node as the new parent node, query the character encapsulation value of the second character from the dictionary table. Traverse the child node data table of the parent node in the sensitive word tree to find the child node corresponding to the character encapsulation value of the second character. If no child node corresponding to the character encapsulation value of the second character is found, the next character is used as the new second character, and step S22 continues. If a child node corresponding to the character encapsulation value of the second character is found, proceed to step S13.

4. The method according to claim 3, characterized in that, Before proceeding to step S22 with the next character designated as the new second character, the method further includes: Determine whether the termination condition for non-continuous sensitive word matching is met. If so, proceed to step S11. The following character is used as the new second character, and the execution of step S22 continues. Further, if it is determined that the termination condition of the non-continuous sensitive word matching is not met, the following character is used as the new second character, and the execution of step S22 continues.

5. The method according to claim 1, characterized in that, Prior to step S11, the method further includes: The text to be detected is preprocessed, including but not limited to: converting non-text characters into fixed placeholders, converting between traditional and simplified Chinese characters, and converting emoticons into text characters.

6. A method for constructing a sensitive word tree, characterized in that, include: Step S31: Create the root node of the sensitive word, use the root node as the initial parent node, and read the first character of the sensitive word as the initial third character; Step S32: Obtain the character encapsulation value of the third character and the character encapsulation value of at least one associated character of the third character from the dictionary table; create multiple child nodes of the parent node; and write the character encapsulation value of the third character and the character encapsulation value of at least one associated character of the third character into the child node data table of the parent node; wherein, in the dictionary table, the character is used as the key value, and the character and its associated pinyin, left and right split characters, and variant character sets are encapsulated together as the character encapsulation value; the variant character set in the character encapsulation value stores the address of the corresponding variant character value in the dictionary table; the at least one associated character of the third character does not include variant characters; Step S33: Determine whether the third character is the last character of the sensitive word. If yes, set the multiple child nodes as leaf nodes; if no, set each child node as the new parent node and the next character of the sensitive word as the new third character, and execute step S32.

7. The method according to claim 6, characterized in that, Prior to step S32, the method further includes: Determine whether the third character is a sensitive word connector; If so, create the connecting child node of the parent node, and create the connecting node corresponding to the sensitive word connector, establish the parent-child relationship between the connecting node and the connecting child node, take the connecting node as the new parent node, take the next character as the new third character, and execute step S32; If not, proceed directly to step S32.

8. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the sensitive word identification method according to any one of claims 1-5, and / or, the executable instruction causes the processor to perform an operation corresponding to the sensitive word tree construction method according to claim 6 or 7.

9. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the sensitive word identification method according to any one of claims 1-5, and / or, the executable instruction causes the processor to perform an operation corresponding to the sensitive word tree construction method according to claim 6 or 7.

Images