A sensitive word matching method and device, electronic equipment and storage medium

Abstract

The application provides a sensitive word matching method and device, electronic equipment and storage medium. The method comprises: performing string matching on a to-be-detected text based on a dictionary tree to obtain a hit sensitive word content; the dictionary tree is constructed according to a sensitive word library, the sensitive word library comprises at least one of an accurate word, a spacing word and a disordered word; nodes in the dictionary tree are associated through hash mapping; and a sensitive word matching result is obtained according to the hit sensitive word content. The to-be-detected text is matched based on the dictionary tree, the nodes in the dictionary tree are associated through hash mapping, and the matching rate is improved. The dictionary tree is constructed according to the sensitive word library, the sensitive word library comprises at least one of the accurate word, the spacing word and the disordered word, and accurate word matching, spacing word matching and disordered word matching of the to-be-detected text are realized. The sensitive word matching is performed in multiple ways based on the dictionary tree, and the accuracy of the sensitive word matching is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a sensitive word matching method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, the rapid development of computer networks and artificial intelligence technologies has led to a significant increase in the speed and scale of information generation and dissemination. In some scenarios, sensitive and high-risk information may emerge. To prevent the unrestricted spread of this information, sensitive word matching and retrieval are typically performed. Current sensitive word matching methods usually employ regular expressions, but this approach may fail to find some deeply forged sensitive words, resulting in low matching accuracy. Summary of the Invention

[0003] The purpose of this application is to provide a sensitive word matching method, apparatus, electronic device, and storage medium that performs string matching on the text to be detected based on a trie and improves the accuracy of sensitive word matching through various methods.

[0004] In a first aspect, embodiments of this application provide a sensitive word matching method, including: matching strings of the text to be detected based on a trie to obtain the content of the hit sensitive words; wherein, the trie is constructed based on a sensitive word library, which includes at least one of precise words, spaced words, and disordered words; the nodes in the trie are associated through a hash mapping; and the sensitive word matching result is obtained based on the content of the hit sensitive words.

[0005] In the above implementation process, string matching is performed on the text to be detected based on a trie. Nodes in the trie are associated through hash mapping to improve the matching speed. The trie is constructed based on a sensitive word library, which includes at least one of precise words, spaced words, and randomized words, enabling precise word matching, spaced word matching, and randomized word matching of the text to be detected. Multiple methods of sensitive word matching based on the trie improve the accuracy of sensitive word matching.

[0006] Optionally, in this embodiment of the application, before matching strings in the text to be detected based on the trie to obtain the content of the hit sensitive words, the method further includes: obtaining a sensitive word library and loading the sensitive word library into memory; wherein, the spacer words are obtained by inserting precise words into other characters, and the disordered words are generated by arranging precise words in disorder; and a trie is generated by hash mapping based on the hash values ​​corresponding to the sensitive words in the sensitive word library.

[0007] In the above implementation process, a sensitive word library is loaded into memory. Based on the hash values ​​corresponding to the sensitive words in the library, a trie is generated through hash mapping. Nodes in the trie are associated through hash mapping, achieving a faster matching rate and improving trie query performance. Furthermore, multiple types of sensitive word libraries can be selected to construct the trie, enabling various query methods and improving the accuracy of sensitive word queries.

[0008] Optionally, in this embodiment, an update request for the trie is received. The update request includes adding a sensitive word, deleting a sensitive word, or modifying a sensitive word. If the update request is to add a sensitive word, a new node corresponding to the added sensitive word is added to the trie, and the added sensitive word is stored in the new node. If the update request is to delete a sensitive word, the delete node corresponding to the sensitive word to be deleted is found in the nodes of the trie, and the attribute of the sensitive word stored in the delete node is marked as deleted. If the update request is to modify a sensitive word, the modify node corresponding to the sensitive word to be modified is found in the nodes of the trie, and the sensitive word stored in the modify node is modified.

[0009] In the above implementation, by performing word update operations at the corresponding nodes, the problem of memory and other resource consumption caused by rebuilding the trie every time the dictionary is updated, as well as the problem of large objects affecting GC (Garbage Collection), is mitigated. This improves resource utilization and trie performance.

[0010] Optionally, in this embodiment of the application, a trie is generated based on the hash values ​​corresponding to sensitive words in the sensitive word library through hash mapping, including: classifying sensitive words in the sensitive word library according to preset rules to obtain a sensitive word library of each type; and generating a trie of the corresponding type based on the sensitive word library of each type.

[0011] In the above implementation process, sensitive words in the sensitive word library are classified, and a corresponding trie is generated based on each type of sensitive word library. This reduces the time spent initializing the complete sensitive word library, reduces the memory requirements of individual hardware device nodes, lowers server storage requirements, and improves the matching speed.

[0012] Optionally, in this embodiment of the application, the string matching of the text to be detected based on the trie to obtain the content of the hit sensitive words includes: obtaining the type of the text to be detected based on the text to be detected; routing the text to be detected to the trie that matches the type of the text to be detected based on the type of the text to be detected, performing string matching of the text to be detected, and obtaining the content of the hit sensitive words.

[0013] In the above implementation process, the text to be detected is routed to a trie that matches the type of the text to be detected for matching detection, reducing the time-consuming matching process. Furthermore, some words may have different meanings in different application scenarios; that is, they may be sensitive words in some application scenarios but not in others. Therefore, matching is performed in the trie corresponding to the type of the text to be detected, reducing the false detection rate and improving the accuracy of sensitive word matching.

[0014] Optionally, in this embodiment of the application, the string matching of the text to be detected based on the trie to obtain the content of the hit sensitive words includes: traversing the string in the text to be detected in the trie, and if a sensitive word in the sensitive word library is matched within a preset character interval, the content of the hit sensitive words is obtained, and the content of the hit sensitive words is stored in a doubly queue.

[0015] In the above implementation process, if a sensitive word from the sensitive word library is matched within a preset character interval, the matched sensitive word content is obtained, achieving interval matching of sensitive words. Even if other characters are inserted into the sensitive word, a match can still be made, improving the accuracy of sensitive word matching. Furthermore, a doubly queue is used to store the matched sensitive words, improving the stability of the trie.

[0016] Optionally, in this embodiment of the application, the sensitive word matching result includes the type of sensitive word hit; obtaining the sensitive word matching result based on the content of the hit sensitive word includes: obtaining the matching content in the node corresponding to the hit sensitive word based on the content of the hit sensitive word; comparing the content of the hit sensitive word and the matching content to obtain the type of sensitive word hit; the type of sensitive word hit includes exact matching, interval matching or random matching.

[0017] In the above implementation process, the type of the hit sensitive word is obtained based on the content of the hit sensitive word and the matching content. Not only can the content of the hit sensitive word be obtained, but the matching type can also be further analyzed so as to optimize the trie according to the matching type and improve the performance of the trie.

[0018] Optionally, in this embodiment of the application, after matching strings in the text to be detected based on a trie to obtain the content of the hit sensitive words, the method further includes: determining whether the content of the hit sensitive words is an ignored sensitive word according to a preset ignored word library; if the content of the hit sensitive words is an ignored sensitive word, obtaining the text segment where the ignored sensitive words are located in the detected text; and determining whether the content of the hit sensitive words is effectively hit according to the text segment where each ignored sensitive word is located.

[0019] In the above implementation process, by determining whether the content of the hit sensitive word is an ignored sensitive word, and by determining whether the content of the hit sensitive word is effectively hit based on the text segment where each ignored sensitive word is located, invalid hit sensitive words are excluded, thereby further improving the accuracy of sensitive word matching.

[0020] Optionally, in this embodiment of the application, the method further includes: obtaining the attributes of the sensitive words corresponding to the content of the hit sensitive words; wherein, the attributes of the sensitive words are pre-stored in the trie corresponding to the sensitive word library; if the attribute of the sensitive word is a draft word attribute, then the sensitive word matching result corresponding to the content of the hit sensitive words is sent to the verification queue through an asynchronous message so that the verification queue can verify the sensitive word matching result.

[0021] In the above implementation process, by setting the draft word attribute for sensitive words, the sensitive words with the matched draft word attribute are validated, thereby reducing the false detection rate of sensitive word matching and improving the matching accuracy.

[0022] Secondly, embodiments of this application also provide a sensitive word matching device, including: a matching module, used to perform string matching on the text to be detected based on a trie to obtain the content of the hit sensitive words; wherein, the trie is constructed according to a sensitive word library, and the sensitive word library includes at least one of precise words, spaced words and disordered words; the nodes in the trie are associated through a hash mapping; and a result obtaining module, used to obtain the sensitive word matching result based on the content of the hit sensitive words.

[0023] Thirdly, embodiments of this application also provide an electronic device, including: a processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method described above.

[0024] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the methods described above.

[0025] This application provides a sensitive word matching method, apparatus, electronic device, and storage medium. Based on a trie, it performs string matching on the text to be detected. Nodes in the trie are associated through hash mapping, improving the matching speed. The trie is constructed based on a sensitive word library, which includes at least one of precise words, spaced words, and disordered words, enabling precise word matching, spaced word matching, and disordered word matching of the text to be detected. Multiple sensitive word matching methods based on the trie improve the accuracy of sensitive word matching. By updating words at corresponding nodes, it mitigates the memory and other resource consumption caused by reconstructing the trie, improving resource utilization and the performance of the trie. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart illustrating a sensitive word matching method provided in the first embodiment of this application;

[0028] Figure 2 A schematic diagram of the trie structure provided in the embodiments of this application;

[0029] Figure 3 A flowchart illustrating the sensitive word matching process provided for the second embodiment of this application;

[0030] Figure 4 This is a schematic diagram of the sensitive word matching device provided in the embodiments of this application;

[0031] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.

[0034] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0035] Please see Figure 1The illustration shows a flowchart of a sensitive word matching method provided in the first embodiment of this application. The sensitive word matching method provided in this embodiment can be applied to electronic devices, which may include a terminal and a server; wherein the terminal may specifically be a smartphone, tablet computer, computer, personal digital assistant (PDA), etc.; the server may specifically be an application server or a web server. The sensitive word matching method may include the following steps:

[0036] Step S110: Based on the trie, perform string matching on the text to be detected to obtain the content of the hit sensitive words; wherein, the trie is constructed according to the sensitive word library, which includes at least one of precise words, spaced words and disordered words; the nodes in the trie are associated through hash mapping.

[0037] Step S120: Obtain the sensitive word matching results based on the content of the hit sensitive words.

[0038] In step S110: A trie, also known as a word lookup tree or trie, is a tree-like structure. The trie is constructed based on strings corresponding to a sensitive word database. It includes a root node and leaf nodes. The root node does not store characters, while each leaf node stores one character. The characters along the path from the root node to a given node, when concatenated, form the string corresponding to that node. This string can consist of multiple characters or a single character. Tries can be used to count, sort, and store large numbers of strings. Operations on a trie can include searching, inserting, and deleting. In this embodiment, the nodes in the trie are associated through a hash mapping.

[0039] The text to be detected is text that needs to be matched for sensitive words, such as text that may contain malicious, unhealthy, or uncivilized language.

[0040] A sensitive word database can be a database composed of multiple sensitive words, including at least one of precise words, spacer words, and randomized words. Precise words are the characters corresponding to the sensitive words and do not contain any other characters unrelated to them. Spacer words are created by inserting other characters into precise words; these other characters can be characters, words, symbols, etc. For example, inserting " / / " into the precise word "AB" would result in the spacer word "A / / B". Randomized words are generated by scrambling the order of the precise words; for example, reversing the order of the characters in the precise word "AB" would result in the randomized word "BA".

[0041] The text to be detected is segmented into strings. Each string in the text is then used to sequentially traverse a pre-generated trie to obtain the matched sensitive words in the trie's sensitive word database. These matched sensitive words can be obtained by matching precise words, spaced words, or random words, thus enabling precise word matching, spaced word matching, and random word matching of the text to be detected.

[0042] In step S120: the sensitive word matching result may include the type of sensitive word hit and whether the hit is valid. The type of sensitive word hit is obtained based on the content of the hit sensitive words in the text to be detected and the matching content of the corresponding sensitive words. The validity of the hit sensitive words can also be determined based on the content of the hit sensitive words in the text to be detected and a preset ignore word list.

[0043] In the above implementation process, string matching is performed on the text to be detected based on a trie. Nodes in the trie are associated through hash mapping to improve the matching speed. The trie is constructed based on a sensitive word library, which includes at least one of precise words, spaced words, and randomized words, enabling precise word matching, spaced word matching, and randomized word matching of the text to be detected. Multiple methods of sensitive word matching based on the trie improve the accuracy of sensitive word matching.

[0044] Optionally, in this embodiment of the application, before matching strings in the text to be detected based on the trie to obtain the content of the hit sensitive words, the method further includes: obtaining a sensitive word library and loading the sensitive word library into memory; wherein, the spacer words are obtained by inserting precise words into other characters, and the disordered words are generated by arranging precise words in disorder; and a trie is generated by hash mapping based on the hash values ​​corresponding to the sensitive words in the sensitive word library.

[0045] In the specific implementation process: the obtained sensitive word library is loaded into memory. Based on the hash values ​​corresponding to the sensitive words in the sensitive word library, the hash values ​​corresponding to the sensitive words are mapped to the nodes of the trie through a hash map (HashMap), thus generating the trie in memory. By using the hash values ​​of the sensitive words, an O(1) matching rate is achieved, improving the trie query performance.

[0046] For example, we can use a HashMap to simulate the implementation of a node class. Each key in the hash table represents a letter, and its corresponding value points to a new HashMap. First, we initialize a HashMap as the root node of the trie. During the insertion of a string, we traverse each character of the string and start searching from the root node. If the current Map returns true when we use containsKey() to search for that character, we use the get() keyword to enter the Map retrieved for that character and continue searching for the next character. Otherwise, we create a new HashMap, use that character as the key, and put the new hash table into the current Map. By inserting characters in this way multiple times, we can build a trie.

[0047] Please see Figure 2 The diagram shown is a trie structure provided in an embodiment of this application.

[0048] If the sensitive word list includes the six words b, abc, abd, bcd, abcd, efg, and hii, then the trie created based on the sensitive word list will look like this: Figure 2 As shown. The root node contains no characters, and each child node (excluding the root node) contains one character. The characters along the path from the root node to any given node, when concatenated, form the string corresponding to that node. All child nodes of each node contain distinct characters.

[0049] During implementation, a flag is set in the node structure to indicate whether the node constitutes a word or keyword. For example, the words "abc" and "abcd" will be marked as words at node "c" and node "d" respectively, thus implementing the word inclusion function.

[0050] It should be noted that the sensitive word library used to construct the trie includes at least one of the following: precise words, spaced words, and randomized words. That is, a trie can be constructed using precise words, spaced words, and randomized words, or any one or two of these can be selected. For example, if only randomized words are chosen to construct the trie, this can be done by scrambling the characters of the sensitive words in the sensitive word library to obtain a randomized word library, and then building the trie based on this randomized word library.

[0051] In the above implementation process, a sensitive word library is loaded into memory. Based on the hash values ​​corresponding to the sensitive words in the library, a trie is generated through hash mapping. Nodes in the trie are associated through hash mapping, achieving a faster matching rate and improving trie query performance. Furthermore, multiple types of sensitive word libraries can be selected to construct the trie, enabling various query methods and improving the accuracy of sensitive word queries.

[0052] Optionally, in this embodiment, an update request for the trie is received. The update request includes adding a sensitive word, deleting a sensitive word, or modifying a sensitive word. If the update request is to add a sensitive word, a new node corresponding to the added sensitive word is added to the trie, and the added sensitive word is stored in the new node. If the update request is to delete a sensitive word, the delete node corresponding to the sensitive word to be deleted is found in the nodes of the trie, and the attribute of the sensitive word stored in the delete node is marked as deleted. If the update request is to modify a sensitive word, the modify node corresponding to the sensitive word to be modified is found in the nodes of the trie, and the sensitive word stored in the modify node is modified.

[0053] In the specific implementation process: After the trie is built, if the sensitive word database changes, such as adding, deleting, or modifying sensitive words, the trie needs to be updated. The system receives update requests for the trie, which include update operations such as adding, deleting, or modifying sensitive words. The system then updates the trie according to the update requests.

[0054] As an optional implementation, after receiving an update request, the sensitive word matching system can broadcast the notification to the nodes in the trie in real time, and the corresponding nodes can perform update operations according to the update request.

[0055] For example, if the update request is to add a sensitive word, the sensitive word is broken down into its corresponding characters, and a new node corresponding to the sensitive word is added to the trie. Continuing with the above embodiment, if the sensitive word is "e", and the "e" node already exists in the trie, it can be directly marked as a word, thus adding the sensitive word "e". If the sensitive word is "efgh", and there is no corresponding node in the current trie, an "h" edge and its corresponding node can be added after the "efg" node, and this newly added node can be marked as a word, thus adding the sensitive word "efgh".

[0056] If the update request is to delete sensitive words, there's no need to rebuild the trie. Simply find the delete node corresponding to the sensitive word in the trie and mark the attribute of the sensitive word stored in that node as deleted. It's important to note that the delete node itself is not directly deleted; rather, the attribute of the sensitive word stored in the delete node is marked as deleted. This avoids situations where the delete node is an intermediate node or a prefix path node of other words, which could lead to errors in related terms if deleted directly.

[0057] If the update request is to modify sensitive words, then in the trie, find the modification node corresponding to the sensitive word that needs to be modified, and modify the sensitive word stored in the modification node.

[0058] In the above implementation, by performing word update operations at the corresponding nodes, the problem of excessive memory and other resource consumption caused by rebuilding the trie every time the dictionary is updated, and the impact of large objects generated by rebuilding the trie on GC (Garbage Collection), is mitigated. This improves resource utilization and trie performance.

[0059] Optionally, in this embodiment of the application, a trie is generated based on the hash values ​​corresponding to sensitive words in the sensitive word library through hash mapping, including: classifying sensitive words in the sensitive word library according to preset rules to obtain a sensitive word library of each type; and generating a trie of the corresponding type based on the sensitive word library of each type.

[0060] In the specific implementation process: Due to the massive number of sensitive words in the current sensitive word database, initializing the trie takes a long time. Therefore, sensitive words in the database can be classified according to preset rules to obtain a sensitive word database of each type. These preset rules can be application scenarios or sensitive word matching types. Taking application scenarios as an example, application scenarios include malicious language and unhealthy connotations. Based on different application scenarios, the sensitive words in the database are classified to obtain a sensitive word database for each application scenario. Based on each type of sensitive word database, a corresponding type of trie is generated, meaning the type of the trie matches the classification type of the sensitive words.

[0061] As one implementation method, different types of sensitive word libraries can be loaded onto different hardware device nodes. By storing different types of sensitive word libraries on different hardware device nodes, the memory requirements of a single hardware device node can be reduced, and the server storage configuration and initialization time can be reduced.

[0062] In the above implementation process, sensitive words in the sensitive word library are classified, and a corresponding trie is generated based on each type of sensitive word library. This reduces the time spent initializing the complete sensitive word library, reduces the memory requirements of individual hardware device nodes, lowers server storage requirements, and improves the matching speed.

[0063] Optionally, in this embodiment of the application, the string matching of the text to be detected based on the trie to obtain the content of the hit sensitive words includes: obtaining the type of the text to be detected based on the text to be detected; routing the text to be detected to the trie that matches the type of the text to be detected based on the type of the text to be detected, performing string matching of the text to be detected, and obtaining the content of the hit sensitive words.

[0064] In the specific implementation process: Based on the text to be detected, the type of the text to be detected is obtained, and the type of the text to be detected is matched with the type of the trie to obtain the type of the trie that matches the type of the text to be detected; The text to be detected is routed to the target hardware device node corresponding to the trie that matches the type of the text to be detected, and the target hardware device node starts the trie to complete the matching of the text string to be detected and obtain the content of the hit sensitive words.

[0065] For example, if the application scenario type of the text to be detected is found to be unhealthy color, then the text to be detected is routed to the target hardware device node that handles the matching of sensitive words for unhealthy color, the trie for matching sensitive words for unhealthy color is started, and the text string of the text to be detected is matched to obtain the content of the hit sensitive words.

[0066] In an optional embodiment, if there are multiple texts to be detected at the same time, the type of each text to be detected can be obtained, and each text to be detected can be routed to a trie that matches the type of the text to be detected. The corresponding trie performs concurrent matching to obtain the content of the hit sensitive words corresponding to each text to be detected, thereby reducing the matching time.

[0067] In the above implementation process, the text to be detected is routed to a trie that matches the type of the text to be detected for matching detection, reducing the time-consuming matching process. Furthermore, some words may have different meanings in different application scenarios; that is, they may be sensitive words in some application scenarios but not in others. Therefore, matching is performed in the trie corresponding to the type of the text to be detected, reducing the false detection rate and improving the accuracy of sensitive word matching.

[0068] Optionally, in this embodiment of the application, the string matching of the text to be detected based on the trie to obtain the content of the hit sensitive words includes: traversing the string in the text to be detected in the trie, and if a sensitive word in the sensitive word library is matched within a preset character interval, the content of the hit sensitive words is obtained, and the content of the hit sensitive words is stored in a doubly queue.

[0069] In the specific implementation process: the string in the text to be detected is traversed in the trie according to the depth query. If the first character of the sensitive word is hit, it will be determined whether the subsequent characters in the text to be detected within the preset character interval hit the second character of the sensitive word.

[0070] For example, with a preset character interval of 3, the string to be detected is "b%&c", and the sensitive word in the sensitive word library is "bc". The first string "b" in the text to be detected matches the first "b" in the sensitive word "bc"; the second string "%" does not match any character in the sensitive word; the third string "&" does not match any character in the sensitive word; and the fourth string "c" matches the second "c" in the sensitive word "bc". In other words, "b%&c" in the text to be detected matches a sensitive word in the sensitive word library within two character intervals, that is, within the preset character interval. Therefore, "b%&c" in the text to be detected is considered a hit sensitive word.

[0071] The sensitive word matched by "b%&c" is "bc".

[0072] It should be noted that in interval word matching, the characters separating the precise words can be any characters. That is, in the sensitive word "bc", the characters separating the words can be any characters, as long as the number of characters is within the preset interval number, it can be matched.

[0073] Detected sensitive words are stored in a double-ended queue (DEX). A DEX is a data structure where both ends can be operated on (dequeue and enqueue). This prevents word disorder during dynamic matching caused by updates such as adding, deleting, and modifying sensitive words in the trie. In other words, by storing the detected sensitive words in a DEX, updates such as adding, deleting, and modifying sensitive words are unaffected during trie traversal and lookup, and the words in the trie are not disordered.

[0074] In the above implementation process, if a sensitive word from the sensitive word library is matched within a preset character interval, the matched sensitive word content is obtained, achieving interval matching of sensitive words. Even if other characters are inserted into the sensitive word, a match can still be made, improving the accuracy of sensitive word matching. Furthermore, a doubly queue is used to store the matched sensitive words, improving the stability of the trie.

[0075] Optionally, in this embodiment of the application, the sensitive word matching result includes the type of sensitive word hit; obtaining the sensitive word matching result based on the content of the hit sensitive word includes: obtaining the matching content in the node corresponding to the hit sensitive word based on the content of the hit sensitive word; comparing the content of the hit sensitive word and the matching content to obtain the type of sensitive word hit; the type of sensitive word hit includes exact matching, interval matching or random matching.

[0076] In the specific implementation process: the sensitive word library includes at least one of precise words, spaced words, and randomized words. Therefore, the text to be detected can be matched using precise words, spaced words, and randomized words. The matching content is pre-stored in the nodes corresponding to the matched sensitive words, and precise words, spaced words, and randomized words each have corresponding matching content. If the matched content of a matched sensitive word matches the matching content corresponding to a precise word, the matched sensitive word type is precise matching; if the matched content of a matched sensitive word matches the matching content corresponding to a spaced word, the matched sensitive word type is spaced matching; if the matched content of a matched sensitive word matches the matching content corresponding to a randomized word, the matched sensitive word type is randomized matching.

[0077] Following the above embodiment, the sensitive word content "b%&c" is hit, and the corresponding sensitive word is "bc". The matching content is obtained in the node corresponding to the sensitive word "bc". The sensitive word content and the matching content are compared. If the sensitive word content and the matching content corresponding to the interval word match, it means that the sensitive word hit type is interval matching, that is, the text to be detected hits the sensitive word through interval matching.

[0078] In the above implementation process, the type of the hit sensitive word is obtained based on the content of the hit sensitive word and the matching content. Not only can the content of the hit sensitive word be obtained, but the matching type can also be further analyzed so as to optimize the trie according to the matching type and improve the performance of the trie.

[0079] Optionally, in this embodiment of the application, after matching strings in the text to be detected based on a trie to obtain the content of the hit sensitive words, the method further includes: determining whether the content of the hit sensitive words is an ignored sensitive word according to a preset ignored word library; if the content of the hit sensitive words is an ignored sensitive word, obtaining the text segment where the ignored sensitive words are located in the detected text; and determining whether the content of the hit sensitive words is effectively hit according to the text segment where each ignored sensitive word is located.

[0080] In the specific implementation process: the ignore dictionary is a pre-set list of potentially invalid sensitive words, such as certain sensitive words that need to be filtered, which may be invalid even if they are hit. Upon obtaining the content containing the hit sensitive word, it can be determined whether the hit sensitive word is indeed an ignored sensitive word based on the ignore dictionary. If so, the text segment containing the ignored sensitive word in the detected text is obtained. This text segment can include the start and end positions of the text.

[0081] If all the matched sensitive words are within the text segment containing the ignored sensitive words (i.e., from the beginning to the end of the text segment, and no other paragraphs contain matched sensitive words), then the matched sensitive words are invalid and can be ignored. If the matched sensitive words exist outside the text segment containing the ignored sensitive words, then the matched sensitive words are valid and can be stored.

[0082] In the above implementation process, by determining whether the content of the hit sensitive word is an ignored sensitive word, and by determining whether the content of the hit sensitive word is effectively hit based on the text segment where each ignored sensitive word is located, invalid hit sensitive words are excluded, thereby further improving the accuracy of sensitive word matching.

[0083] Optionally, in this embodiment of the application, the method further includes: obtaining the attributes of the sensitive words corresponding to the content of the hit sensitive words; wherein, the attributes of the sensitive words are pre-stored in the trie corresponding to the sensitive word library; if the attribute of the sensitive word is a draft word attribute, then the sensitive word matching result corresponding to the content of the hit sensitive words is sent to the verification queue through an asynchronous message so that the verification queue can verify the sensitive word matching result.

[0084] In the specific implementation process: the attributes of sensitive words are pre-stored in the trie corresponding to the sensitive word library. For example, some newly added words in the sensitive word library may be falsely detected on the first use. To avoid a large number of false detections, these words with high uncertainty can be marked as draft words. Among them, precise words, spaced words, and disordered words can all be set with the draft word attribute.

[0085] For example, if the sensitive word content "b%&c" is matched, and the corresponding sensitive word is "bc", and the attribute of the sensitive word "bc" is a draft word attribute, then the sensitive word matching result corresponding to the sensitive word content "b%&c" is sent to the verification queue via an asynchronous message. The verification queue then verifies the sensitive word matching result. For example, the verification queue performs draft calculations on the sensitive word "bc" with the draft word attribute, verifying the hit rate and false positive rate corresponding to the sensitive word "bc". Based on the verification results, the sensitive words in the sensitive word library are adjusted. If the false positive rate is compared with the corresponding threshold, and the false positive rate is greater than the threshold, then the matching effect of the sensitive word is considered poor, and it can be deleted from the sensitive word library.

[0086] In the above implementation process, by setting the draft word attribute for sensitive words, the sensitive words with the matched draft word attribute are validated, thereby reducing the false detection rate of sensitive word matching and improving the matching accuracy.

[0087] In an optional embodiment, if the length of the text to be detected exceeds a preset length threshold, the text can be segmented according to a preset length, for example, into segments of 1000 fields each. Concurrent threads can then be used to perform string matching on the segmented text. Furthermore, to avoid sensitive words at or before / after the truncation point, the segments can be interleaved with a preset number of characters before and after the truncation point, thus preventing the problem of missing sensitive words due to truncation.

[0088] Please see Figure 3 The diagram shows a flowchart of sensitive word matching provided in the second embodiment of this application.

[0089] In an optional embodiment, a sensitive word library is obtained and loaded into memory; based on the hash values ​​corresponding to the sensitive words in the sensitive word library, a trie is generated through hash mapping, which can construct a trie for precise words, spaced words, and disordered words.

[0090] The system retrieves the text to be detected and determines whether the text content is empty. If it is empty, the current sensitive word matching process ends directly. If the text content is not empty, the system retrieves the scene information of the text to be detected. The system pre-configures the corresponding matching process for each scene, such as whether the scene enables pinyin matching, interval matching, and random order matching.

[0091] After obtaining the scene information of the text to be detected, fuzzy word processing can be performed on the text. Then, it is determined whether pinyin matching is enabled in the scene. If pinyin matching is enabled, it can be performed in the trie constructed from pinyin sensitive words; otherwise, the process proceeds to the next step. Next, it is determined whether interval matching is enabled in the scene. If interval matching is enabled, it can be performed in the trie constructed from interval sensitive words; otherwise, the process proceeds to the next step. Then, it is determined whether random order matching is enabled in the scene. If random order matching is enabled, it can be performed in the trie constructed from random order sensitive words; otherwise, the process proceeds to the next step. For matched sensitive words, ignore word processing is performed, and the validity of the matched sensitive words is determined. Then, the matched sensitive words and their types are counted, and the matched sensitive words are filtered according to the word application. Finally, a POST request is performed to submit the statistical data to the specified resource for processing, completing the sensitive word matching.

[0092] Please see Figure 4 The diagram shown is a structural schematic of a sensitive word matching device provided in an embodiment of this application; this embodiment of the application provides a sensitive word matching device 200, including:

[0093] The matching module 210 is used to perform string matching on the text to be detected based on a trie to obtain the content of the hit sensitive words; wherein, the trie is constructed according to a sensitive word library, which includes at least one of precise words, spaced words and disordered words; the nodes in the trie are associated through a hash mapping;

[0094] The results module 220 is used to obtain sensitive word matching results based on the content of the hit sensitive words.

[0095] Optionally, in this embodiment of the application, the sensitive word matching device further includes a trie module, used to obtain the sensitive word library and load the sensitive word library into memory; wherein, the interval word is obtained by inserting the precise word into other characters, and the disordered word is generated by arranging the precise word in disorder; the trie is generated by hash mapping based on the hash value corresponding to the sensitive word in the sensitive word library.

[0096] Optionally, in this embodiment of the application, the sensitive word matching device further includes an update module, configured to receive an update request for the trie, the update request including adding a sensitive word, deleting a sensitive word, or modifying a sensitive word; if the update request is to add a sensitive word, then a new node corresponding to the added sensitive word is added to the trie, and the added sensitive word is stored in the new node; if the update request is to delete a sensitive word, then in the nodes of the trie, the deletion node corresponding to the sensitive word to be deleted is searched, and the attribute of the sensitive word stored in the deletion node is marked as deleted; if the update request is to modify a sensitive word, then in the nodes of the trie, the modification node corresponding to the sensitive word to be modified is searched, and the sensitive word stored in the modification node is modified.

[0097] Optionally, in this embodiment of the application, the sensitive word matching device and the trie module are further configured to classify the sensitive words in the sensitive word library based on preset rules to obtain a sensitive word library of each type; and to generate a trie of the corresponding type based on the sensitive word library of each type.

[0098] Optionally, in this embodiment of the application, the sensitive word matching device and the matching module are further configured to obtain the type of the text to be detected based on the text to be detected; based on the type of the text to be detected, route the text to be detected to a trie that matches the type of the text to be detected, perform string matching on the text to be detected, and obtain the content of the hit sensitive words.

[0099] Optionally, in this embodiment of the application, the sensitive word matching device and the matching module are further configured to traverse the string in the text to be detected in the trie, and if a sensitive word in the sensitive word library is matched within a preset character interval, the content of the hit sensitive word is obtained and the content of the hit sensitive word is stored in a doubly queued.

[0100] Optionally, in this embodiment of the application, the sensitive word matching result includes the type of sensitive word hit; the sensitive word matching device and the matching module are further configured to obtain matching content in the node corresponding to the hit sensitive word based on the content of the hit sensitive word; compare the content of the hit sensitive word and the matching content to obtain the type of sensitive word hit; the type of sensitive word hit includes exact matching, interval matching or random matching.

[0101] Optionally, in this embodiment of the application, the sensitive word matching device and the ignored word module are used to determine whether the hit sensitive word content is an ignored sensitive word according to a preset ignored word library; if the hit sensitive word content is an ignored sensitive word, then obtain the text segment in the detected text where the ignored sensitive word is located; and determine whether the hit sensitive word content is effectively hit according to the text segment where each ignored sensitive word is located.

[0102] Optionally, in this embodiment of the application, the sensitive word matching device further includes a draft module, used to obtain the attributes of the sensitive words corresponding to the hit sensitive word content; wherein, the attributes of the sensitive words are pre-stored in the trie corresponding to the sensitive word library; if the attributes of the sensitive words are draft word attributes, the sensitive word matching result corresponding to the hit sensitive word content is sent to the verification queue through an asynchronous message, so that the verification queue verifies the sensitive word matching result.

[0103] It should be understood that this device corresponds to the above-described sensitive word matching method embodiment and is capable of performing the various steps involved in the above method embodiment. The specific functions of this device can be found in the description above, and detailed descriptions are omitted here to avoid repetition. The device includes at least one software functional module that can be stored in memory or embedded in the device's operating system (OS) in the form of software or firmware.

[0104] Please see Figure 5 The diagram shows a structural schematic of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes a processor 310 and a memory 320. The memory 320 stores machine-readable instructions executable by the processor 310. When the machine-readable instructions are executed by the processor 310, the method described above is performed.

[0105] This application also provides a storage medium storing a computer program, which is executed by a processor to perform the above-described method.

[0106] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0107] It should be understood that the disclosed apparatus and methods can also be implemented in other ways, given the several embodiments provided in this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0108] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0109] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A sensitive word matching method, characterized in that, include: Based on a trie, string matching is performed on the text to be detected to obtain the content of the hit sensitive words; wherein, the trie is constructed according to a sensitive word library, which includes at least one of precise words, spaced words, and disordered words; the nodes in the trie are associated through hash mapping; wherein, hash mapping is used to simulate the implementation of node classes, and each key of the hash table represents a letter, and the corresponding value points to a new hash mapping; Based on the content of the hit sensitive words, obtain the sensitive word matching results; Before performing string matching on the text to be detected based on the trie to obtain the content containing the hit sensitive words, the method further includes: The sensitive word library is obtained and loaded into memory; wherein, the spacer word is obtained by inserting other characters into the precise word, and the disordered word is generated by arranging the precise word in a disordered order; Based on the hash values ​​corresponding to the sensitive words in the sensitive word database, the trie is generated through hash mapping; Based on the hash values ​​corresponding to sensitive words in the sensitive word database, the trie is generated through hash mapping, including: The sensitive words in the sensitive word library are classified according to preset rules to obtain a sensitive word library of each type; Based on each type of sensitive word database, generate the corresponding trie; If there are multiple texts to be detected at the same time, the type of each text to be detected is obtained, and each text to be detected is routed to a trie that matches the type of the text to be detected, and concurrent matching is performed by the corresponding trie. The method further includes: receiving an update request for the trie; if the update request is to delete a sensitive word, then in the nodes of the trie, finding the deletion node corresponding to the sensitive word to be deleted, and marking the attribute of the sensitive word stored in the deletion node as deleted; Obtain the attributes of the sensitive words corresponding to the hit sensitive word content; wherein, the attributes of the sensitive words are pre-stored in the trie corresponding to the sensitive word library; if the attribute of the sensitive word is a draft word attribute, then the sensitive word matching result corresponding to the hit sensitive word content is sent to the verification queue through an asynchronous message so that the verification queue can verify the sensitive word matching result; If the length of the text to be detected exceeds a preset length threshold, the text to be detected is segmented according to the preset length, and concurrent threads are used to perform string matching on the segmented text to be detected; when segmenting the text to be detected by length, the segments include a preset number of characters before and after the truncation point.

2. The method according to claim 1, characterized in that, The method further includes: Receive update requests for the trie, the update requests including adding sensitive words, deleting sensitive words, or modifying sensitive words; If the update request is for adding a sensitive word, then a new node corresponding to the new sensitive word is added to the trie, and the new sensitive word is stored in the new node; If the update request is to modify a sensitive word, then in the nodes of the trie, the modification node corresponding to the sensitive word that needs to be modified is found, and the sensitive word stored in the modification node is modified.

3. The method according to claim 1, characterized in that, Based on a trie, string matching is performed on the text to be detected to obtain the content containing sensitive words, including: Based on the text to be detected, the type of the text to be detected is obtained; Based on the type of the text to be detected, the text to be detected is routed to a trie that matches the type of the text to be detected, and the text to be detected is matched with strings to obtain the content of the hit sensitive words.

4. The method according to claim 1, characterized in that, The process of matching strings in the text to be detected based on a trie to obtain the content containing sensitive words includes: The string in the text to be detected is traversed in the trie. If a sensitive word in the sensitive word library is matched within a preset character interval, the content of the hit sensitive word is obtained and stored in a doubly queue.

5. The method according to claim 1, characterized in that, The sensitive word matching results include the type of sensitive word matched. The process of obtaining sensitive word matching results based on the content of the hit sensitive words includes: Based on the content of the hit sensitive words, obtain the matching content in the node corresponding to the hit sensitive words; The content of the hit sensitive words is compared with the matching content to obtain the type of hit sensitive words; the type of hit sensitive words includes exact match, interval match or random order match.

6. The method according to claim 1, characterized in that, After performing string matching on the text to be detected based on the trie to obtain the content containing the matched sensitive words, the method further includes: Based on a preset ignore word list, determine whether the hit sensitive word content is an ignore sensitive word; If the detected sensitive word content is an ignored sensitive word, then the text segment containing the ignored sensitive word in the detected text is obtained; Based on the text segment containing each of the ignored sensitive words, determine whether the content containing the hit sensitive words is a valid hit.

7. A sensitive word matching device, characterized in that, include: The matching module is used to perform string matching on the text to be detected based on a trie to obtain the content of the hit sensitive words; wherein, the trie is constructed according to a sensitive word library, which includes at least one of precise words, spaced words, and disordered words; the nodes in the trie are associated through hash mapping; The result acquisition module is used to obtain sensitive word matching results based on the content of the hit sensitive words; It also includes a trie module, used to obtain the sensitive word library and load the sensitive word library into memory; wherein, the interval words are obtained by inserting other characters into the precise words, and the disordered words are generated by arranging the precise words in disorder; the trie is generated by hash mapping based on the hash values ​​corresponding to the sensitive words in the sensitive word library; Sensitive words in the sensitive word library are classified according to preset rules to obtain a sensitive word library of each type; a corresponding trie is generated based on the sensitive word library of each type; if there are multiple texts to be detected at the same time, the type of each text to be detected is obtained, and each text to be detected is routed to a trie that matches the type of the text to be detected, and concurrent matching is performed by the corresponding trie. It is also used to receive update requests for the trie. If the update request is to delete a sensitive word, then in the nodes of the trie, the deletion node corresponding to the sensitive word to be deleted is found, and the attribute of the sensitive word stored in the deletion node is marked as deleted. Obtain the attributes of the sensitive words corresponding to the hit sensitive word content; wherein, the attributes of the sensitive words are pre-stored in the trie corresponding to the sensitive word library; if the attribute of the sensitive word is a draft word attribute, then the sensitive word matching result corresponding to the hit sensitive word content is sent to the verification queue through an asynchronous message so that the verification queue can verify the sensitive word matching result; If the length of the text to be detected exceeds a preset length threshold, the text to be detected is segmented according to the preset length, and concurrent threads are used to perform string matching on the segmented text to be detected; when segmenting the text to be detected by length, the segments include a preset number of characters before and after the truncation point.

8. An electronic device, characterized in that, include: A processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when executed by the processor, perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the method as described in any one of claims 1 to 6.

Images