Abnormal text recognition method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an abnormal text recognition method and device, equipment and a storage medium, and can be applied to the technical field of network security, in the method, a to-be-recognized text and a preset prefix tree are obtained, and the prefix tree comprises node links corresponding to a plurality of preset key texts respectively, each node link is used for storing a preset key text and a plurality of reference bytes obtained by encoding the preset key text; aiming at the plurality of segmented words in the to-be-recognized text, respectively executing the following operations: encoding one segmented word to obtain a plurality of target bytes; sequentially traversing a plurality of node links in the prefix tree based on the segmented word and the plurality of target bytes to obtain a matching result; and when the matching result of the plurality of segmented words meets the preset exception judgment condition, determining that the to-be-recognized text is the exceptional text, thereby realizing the recognition of the exceptional text of a large number of characters, and improving the recognition efficiency of the exceptional text.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of network security technology, and in particular, to a method, apparatus, device, and storage medium for identifying abnormal text. Background Art

[0002] At present, there are tens of thousands of web pages suspected of cyber black and gray industries in the cyberspace. They use advanced technical means to carry out activities such as phishing and data leakage that threaten the property and information security of users, seriously disrupting the normal network order. Therefore, identifying abnormal web pages among a large number of web pages has become an important part of combating cyber black and gray industries.

[0003] In general technology, the AC (Aho-Corasick) algorithm is used to identify the crawled web page text based on preset keywords to determine whether the web page is an abnormal web page. However, the traditional AC algorithm is applicable to the English field with a small number of characters and is difficult to process a large number of Chinese characters, thus reducing the efficiency and accuracy of abnormal web page text recognition. Summary of the Invention

[0004] The embodiments of the present invention provide a method, apparatus, device, and storage medium for identifying abnormal text, which are used to improve the efficiency and accuracy of abnormal text recognition.

[0005] On the one hand, the embodiments of the present application provide a method for identifying abnormal text, and the method includes:

[0006] Obtain the text to be identified and a preset prefix tree, where the prefix tree includes: node links corresponding to multiple preset key texts, each node link is used to store a preset key text, and multiple reference bytes obtained by encoding the one preset key text;

[0007] For multiple word segments in the text to be identified, perform the following operations respectively: encode a word segment to obtain multiple target bytes; based on the one word segment and the multiple target bytes, sequentially traverse multiple node links in the prefix tree to obtain a matching result;

[0008] When the matching results of the multiple word segments meet a preset abnormal judgment condition, determine that the text to be identified is abnormal text.

[0009] Optionally, the one preset key text includes multiple characters, and each character corresponds to multiple reference bytes;

[0010] The node link corresponding to the one preset key text includes multiple nodes, and each node stores a character or a reference byte.

[0011] Optionally, each node in the node link corresponds to a failure pointer, and the failure pointer is used to indicate the next traversed node when the matching fails;

[0012] Among them, the failure pointer of the root node of the prefix tree points to the root node;

[0013] For each non-root node in the prefix tree, the failure pointer of the non-root node is determined based on the failure pointer of the parent node of the non-root node.

[0014] Optionally, the failure pointer of the non-root node is determined in the following manner:

[0015] Obtain the target node pointed to by the failure pointer of the parent node of the non-root node;

[0016] If at least one child node of the target node contains a child node identical to the non-root node, set the failure pointer of the non-root node to point to the identical child node;

[0017] If at least one child node of the target node does not contain a child node identical to the non-root node, set the failure pointer of the non-root node to the root node.

[0018] Optionally, the obtaining the matching result by sequentially traversing multiple node links in the prefix tree based on the one word segment and the multiple target bytes includes:

[0019] Split the one word segment into multiple characters, and construct a sequence to be matched based on the multiple characters and the multiple target bytes;

[0020] Sequentially traverse the multiple node links based on the sequence to be matched;

[0021] For each node link, when there is a set of nodes in the node link that match the sequence to be matched, take the fact that the sequence to be matched hits the node link as the matching result.

[0022] Optionally, the taking the fact that the sequence to be matched hits the node link as the matching result when there is a set of nodes in the node link that match the sequence to be matched includes:

[0023] Compare the multiple objects to be matched in the sequence to be matched with the nodes at the corresponding positions in the node link one by one, and when the comparison result meets a preset condition, take the fact that the sequence to be matched hits the node link as the matching result; the preset condition is that the comparison result of each object to be matched is a successful match;

[0024] The comparison process corresponding to each object to be matched includes the following operations:

[0025] Compare the i-th object to be matched in the sequence to be matched with the i-th node in the node link, where i > 0;

[0026] When the i-th object to be matched matches the i-th node and the i-th node is a non-leaf node, compare the (i + 1)-th object to be matched in the sequence to be matched with the (i + 1)-th node in the node link.

[0027] Optionally, it further includes:

[0028] When the i-th object to be matched does not match the i-th node, compare the i-th object to be matched with the node pointed to by the failure pointer associated with the i-th node.

[0029] Optionally, it further includes:

[0030] When the i-th object to be matched matches the i-th node and the i-th node is a leaf node, compare the (i + 1)-th object to be matched with the node pointed to by the failure pointer associated with the i-th node.

[0031] Optionally, the determining that the text to be recognized is an abnormal text when the matching results of the multiple word segmentations meet a preset abnormal judgment condition includes:

[0032] Filter out at least one word segmentation whose matching result is hitting the node link from the multiple word segmentations;

[0033] When the number of the at least one word segmentation is greater than a preset threshold, determine that the text to be recognized is an abnormal text.

[0034] On the one hand, an embodiment of the present application provides an abnormal text recognition device, and the device includes:

[0035] An acquisition module, configured to acquire a text to be recognized and a preset prefix tree, where the prefix tree includes: node links corresponding to multiple preset key texts, each node link is used to store a preset key text, and multiple reference bytes obtained by encoding the one preset key text;

[0036] A processing module, configured to perform the following operations respectively for multiple word segmentations in the text to be recognized: encode a word segmentation to obtain multiple target bytes; based on the one word segmentation and the multiple target bytes, sequentially traverse multiple node links in the prefix tree to obtain a matching result;

[0037] A judgment module, configured to determine that the text to be recognized is an abnormal text when the matching results of the multiple word segmentations meet a preset abnormal judgment condition.

[0038] On the one hand, an embodiment of the present application provides a computer device, including:

[0039] A memory for storing program instructions;

[0040] A processor for calling the program instructions stored in the memory and executing the steps of the above-mentioned abnormal text recognition method according to the obtained program.

[0041] On the one hand, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the program runs on the computer device, the computer is caused to execute the steps of the above-mentioned abnormal text recognition method.

[0042] On the one hand, an embodiment of the present application provides a computer program product including a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer device, the computer device is caused to execute the steps of the above-mentioned abnormal text recognition method.

[0043] In the embodiment of the present application, the node link in the prefix tree is used to save the preset key text and multiple reference bytes obtained by encoding the key text. Compared with directly saving Chinese characters by nodes in the prefix tree, the possibility of width explosion of the prefix tree is reduced, which is equivalent to compressing the width of the prefix tree, thereby improving the efficiency and stability of abnormal text recognition. Secondly, for each word segment of the text to be recognized, the word segment and the corresponding multiple target bytes are sequentially traversed through multiple node links in the prefix tree to obtain a matching result. When the matching results of multiple word segments meet the preset abnormal judgment conditions, it is determined that the text to be recognized is an abnormal text. In this way, the final recognition result of the text to be recognized is determined through the matching results of multiple word segments, improving the accuracy of abnormal text recognition. Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 It is a schematic structural diagram of a system architecture provided by an embodiment of the present application;

[0046] Figure 2 It is a schematic flowchart of an abnormal text recognition method provided by an embodiment of the present application;

[0047] Figure 3A schematic structural diagram of a prefix tree provided by an embodiment of the present application;

[0048] Figure 4 A schematic structural diagram of traversing a prefix tree provided by an embodiment of the present application;

[0049] Figure 5 A schematic structural diagram of an abnormal text recognition device provided by an embodiment of the present application;

[0050] Figure 6 A schematic structural diagram of a computer device provided by an embodiment of the present application. Detailed implementation manners

[0051] In order to make the objectives, technical solutions and beneficial effects of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0052] It should be noted that the brief description of the terms in this application is only for the convenience of understanding the embodiments described next, rather than intending to limit the embodiments of this application. Unless otherwise specified, these terms should be understood in their ordinary and general meanings.

[0053] The terms "first", "second", "third", etc. in the description and claims of this application and the above drawings are used to distinguish similar or like objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that such terms can be interchanged under appropriate circumstances.

[0054] The terms "comprising" and "having" and any variations thereof are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components does not necessarily have to be limited to all the components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0055] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or a combination of hardware or / and software code that can perform functions related to that element.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0057] The following briefly introduces the system architecture diagram applicable to the technical solutions of the embodiments of the present application. It should be noted that the following introduced processes are only used to illustrate the embodiments of the present application rather than to limit them.

[0058] Refer to Figure 1 , which is a system architecture diagram applicable to the embodiments of the present application. The system architecture at least includes a terminal device 101 and a server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. The present application does not make specific limitations on the number of terminal devices 101 and servers 102.

[0059] The terminal device 101 is pre-installed with an application having an abnormal text recognition function. The application is a client application, a web application, a mini-program application, etc. The terminal device 101 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart home appliance, a smart voice interaction device, a smart vehicle-mounted device, etc., but is not limited thereto.

[0060] The server 102 is the background server of the application. The server 102 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms, but is not limited thereto.

[0061] It should be noted that the method in the embodiments of the present application can be executed independently by the terminal device 101 or the server 102, or can be jointly executed by the terminal device 101 and the server 102.

[0062] In the embodiments of the present application, the terminal device 101 and the server 102 can be directly or indirectly communicatively connected through one or more networks. The network can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network. Of course, it can also be other possible networks, and the embodiments of the present application do not limit this.

[0063] Based on the Figure 1 system architecture diagram shown below, the embodiments of the present application provide a process of an abnormal text recognition method. The process of this method can be executed by the Figure 1 terminal device 101 shown below, or can be executed by the server 102, or can be executed by the interaction between the terminal device 101 and the server 102. As Figure 2 shown below, it includes the following steps:

[0064] Step 201, obtain the text to be recognized and a preset prefix tree. The prefix tree includes: node links corresponding to multiple preset key texts, each node link is used to store a preset key text, and multiple reference bytes obtained by encoding a preset key text.

[0065] Specifically, the text to be recognized can be web page text obtained by crawling a web page through web crawler technology, or can be text obtained in other scenarios.

[0066] The prefix tree, also known as the dictionary tree or the word search tree, is a tree structure that realizes efficient storage and retrieval of a set of strings by sharing common prefixes, and is an indispensable core basis for the AC algorithm (Aho-Corasick algorithm). The AC algorithm realizes the matching of multiple pattern strings by constructing a prefix tree and adding failure pointers to the nodes in the prefix tree. Its key functions such as multi-pattern matching and fast jumping all rely on the construction and expansion of the prefix tree.

[0067] In some embodiments, a preset key text includes multiple characters, and each character corresponds to multiple reference bytes; the node link corresponding to a preset key text includes multiple nodes, and each node stores a character or a reference byte.

[0068] Specifically, the preset key texts are pre-set sensitive and / or abnormal words, and each preset key text contains multiple Chinese characters. Each Chinese character is encoded into 3 reference bytes through the UTF-8 encoding method, namely the high 8-bit byte, the middle 8-bit byte, and the low 8-bit byte. In the order of the Chinese characters in the preset key text, each Chinese character and the corresponding reference bytes are sequentially stored in the nodes of the node link.

[0069] For example, for the first character of the preset key text, the high 8-bit byte of the first character is stored in the first node of the node link corresponding to the preset key text, the middle 8-bit byte of the first character is stored in the second node of the node link corresponding to the preset key text, and the first character itself is stored in the third node of the node link corresponding to the preset key text, wherein the second node is a child node of the first node, and the third node is a child node of the second node; then for the second character, the high 8-bit byte of the second character is stored in the fourth node of the node link corresponding to the preset key text, the middle 8-bit byte of the second character is stored in the fifth node of the node link corresponding to the preset key text, and the second character itself is stored in the sixth node of the node link corresponding to the preset key text, wherein the fourth node is a child node of the third node, the fifth node is a child node of the fourth node, and the sixth node is a child node of the fifth node; and so on, until the last character of the preset key text and the corresponding reference byte are stored in the node link.

[0070] For example, the word "online shopping" is a preset key text. The word "online shopping" is encoded into 6 reference bytes using UTF-8 encoding, so the corresponding node link contains 6 nodes. Among them, the high 8-bit byte of "网" is stored in the first node of the node link, the middle 8-bit byte of "网" is stored in the second node of the node link, the word "网" itself is stored in the third node of the node link, the high 8-bit byte of "购" is stored in the fourth node of the node link, the middle 8-bit byte of "购" is stored in the fifth node of the node link, and the word "购" itself is stored in the sixth node of the node link.

[0071] It should be noted that in the embodiment of the present application, the lower 8-bit byte may not be stored in the node, but may be stored in the node corresponding to other bytes (such as the middle 8-bit byte or the upper 8-bit byte). This application does not make any specific limitations on this.

[0072] Alternatively, Chinese characters may be encoded as two reference bytes using GB2312 and / or GBK encoding, or as two reference bytes or four reference bytes using UTF-16, which is not specifically limited in this application. However, it should be noted that the encoding method of encoding a Chinese character as two reference bytes can compress the vertical height of the prefix tree and reduce the storage resources of the prefix tree, if the number of Chinese characters in the preset key text is the same, thereby further improving the efficiency of abnormal text recognition.

[0073] In the embodiments of the present application, for each node link, the high-order byte of a Chinese character is stored in the first node of the node link, so that there are at most 256 possibilities for the child nodes under a node in the prefix tree. Compared with directly storing tens of thousands of Chinese characters in the nodes, the width of the prefix tree is compressed, and the storage resources of the prefix tree are reduced, so as to improve the speed of traversing the prefix tree in the subsequent matching process and help improve the recognition efficiency of abnormal texts.

[0074] In some embodiments, each node in the node link corresponds to a failure pointer, and the failure pointer is used to indicate the next node to be traversed when the matching fails; wherein, the failure pointer of the root node of the prefix tree points to the root node; for each non-root node in the prefix tree, the failure pointer of the non-root node is determined based on the failure pointer of the parent node of the non-root node.

[0075] In some embodiments, the failure pointer of the non-root node is determined in the following manner: obtain the target node pointed to by the failure pointer of the parent node of the non-root node; if at least one child node of the target node contains a child node identical to the non-root node, set the failure pointer of the non-root node to point to the identical child node; it should be noted that the child node identical to the non-root node here cannot be the non-root node itself.

[0076] If at least one child node of the target node does not contain a child node identical to the non-root node, set the failure pointer of the non-root node to the root node.

[0077] Specifically, in the process of using the prefix tree to match the text to be recognized, if the text to be recognized does not match a certain node in the prefix tree, the algorithm needs to know where to start the next match of the recognized text. Therefore, a failure pointer is configured for each node in the prefix tree to indicate the next node to be traversed in the prefix tree when the matching fails.

[0078] For example, referring to Figure 3 , two node links of the prefix tree are created with the preset key texts "Chinese enterprises" and "enterprises" respectively, where "Chinese enterprises" corresponds to node link 301 and "enterprises" corresponds to node link 302.

[0079] In Figure 3In the prefix tree shown, the failure pointer of the root node (i.e., root) points to the root node itself. For node link 301, the failure pointer of the parent node of node 3011 (i.e., the root node) points to the root node itself, so the root node is the target node; this target node includes two child nodes, node 3011 itself and node 3021, and node 3021 is not the same as node 3011. That is, after excluding node 3011 itself, there is no child node of this target node that is the same as node 3011. Therefore, the failure pointer of node 3011 points to the root node. The failure pointers of nodes 3012 to 3016 are the same and will not be elaborated here.

[0080] The failure pointer of the parent node of node 3017 points to the root node, so the root node is the target node. Among the two child nodes (node 3011 itself and node 3021) of this target node, there is a child node that is the same as node 3017, that is, node 3021. Therefore, set the failure pointer of node 3017 to point to node 3021; similarly, among the child nodes of the target node (node 3021) to which the failure pointer of the parent node of node 3018 (node 3017) points, there is a child node that is the same as node 3018, that is, node 3022. Therefore, set the failure pointer of node 3018 to point to node 3022. The failure pointers of nodes 3019 to 30112 and each node in node link 302 are the same and will not be elaborated here.

[0081] In the embodiments of the present application, a failure pointer is set for each node in the prefix tree, which is used to indicate which node the object to be matched will match next when the text to be recognized fails to match a certain node in the prefix tree. Through the preset jump path of the failure pointer, the repeated calculation during the algorithm operation is alleviated, the calculation time is saved, and no additional memory overhead is added.

[0082] Step 202, for multiple word segments in the text to be recognized, perform the following operations respectively: Encode a word segment to obtain multiple target bytes; Based on a word segment and multiple target bytes, traverse multiple node links in the prefix tree in sequence to obtain a matching result.

[0083] Specifically, when matching the text to be recognized with the prefix tree, the word segments in the text to be recognized are encoded using the same encoding method as the preset key text to obtain multiple target bytes for each word segment. Among them, one character corresponds to multiple target bytes. The encoding method can use the UTF-8 encoding method to encode a Chinese character into 3 target bytes, or use the GB2312 and / or GBK encoding method to encode a Chinese character into 2 target bytes, or use UTF-16 to encode a Chinese character into 2 target bytes or 4 target bytes. In this regard, the present application does not make specific limitations.

[0084] For each segmented word of the text to be recognized, traverse multiple node links in the prefix tree in sequence according to the high and low order of multiple target bytes corresponding to the segmented word, and obtain the matching result of the segmented word. If there is a node link in the prefix tree that matches the segmented word and the corresponding multiple target bytes, it is considered that the matching result of the segmented word hits the node link.

[0085] In some embodiments, a segmented word is split into multiple characters, and a sequence to be matched is constructed based on the multiple characters and multiple target bytes; the multiple node links are traversed in sequence based on the sequence to be matched; for each node link, when there is a node set in the node link that matches the sequence to be matched, the sequence to be matched hitting the node link is used as the matching result.

[0086] Specifically, using the same encoding method as the preset key text, encode the multiple characters split from a segmented word to obtain multiple target bytes for each character. According to the creation rule of the prefix tree, construct the multiple characters and multiple target bytes into a sequence to be matched, so that the format of the sequence to be matched is consistent with the format of the node link in the prefix tree. If there is a node link in the prefix tree that matches the sequence to be matched, it is considered that the sequence to be matched hits the node link in the prefix tree.

[0087] For example, use the UTF-8 encoding method to encode each character in the preset key text, so that each character in the preset key text corresponds to three reference bytes, and the node link of the prefix tree created based on the preset key text and the corresponding reference bytes, that is, the first node of the node link stores the high-order byte of the character, the second node stores the middle-order byte of the character, and the third node stores the character itself. Based on the same encoding method, encode each character of the segmented word in the text to be recognized into three target bytes. Based on the rule of creating the node link in the prefix tree, construct a sequence to be matched based on the high 8-bit byte and the middle 8-bit byte of each character and the corresponding three target bytes.

[0088] In the actual application process, multiple segmented words and multiple target bytes in the text to be recognized form a sequence to be matched. Based on this one sequence to be matched, continuously traverse multiple node links in the prefix tree to improve the flexibility and efficiency of matching.

[0089] In the embodiments of the present application, use the same encoding method as the preset key text to encode multiple segmented words of the text to be recognized, obtain multiple target bytes corresponding to each segmented word, and according to the creation rule of the node link in the prefix tree, construct the segmented word and its corresponding multiple target bytes into a sequence to be matched, so that the sequence to be matched has the same format as the node link in the prefix tree, improving the rate of text recognition.

[0090] In some embodiments, multiple objects to be matched in the sequence to be matched are sequentially compared with the nodes at the corresponding positions in the node link, and when the comparison result meets the preset condition, the node link hit by the sequence to be matched is used as the matching result; the comparison process corresponding to each object to be matched includes the following operations:

[0091] Compare the i-th object to be matched in the sequence to be matched with the i-th node in the node link, where i>0; when the i-th object to be matched matches the i-th node and the i-th node is a non-leaf node, compare the (i + 1)-th object to be matched in the sequence to be matched with the (i + 1)-th node in the node link.

[0092] Specifically, the preset condition is that the comparison result of each object to be matched is a successful match. Once an object to be matched matches a node in the node link, record the position of the object to be matched based on the index of the sequence to be matched.

[0093] In the process of matching the sequence to be matched and the node link, when the i-th object to be matched in the sequence to be matched matches the i-th node in the node link and the i-th node is a non-leaf node, it means that the traversal of the node link has not ended, but the i-th object to be matched has already been successfully matched. Therefore, compare the (i + 1)-th object to be matched with the (i + 1)-th node in the node link to determine whether the sequence to be matched can be successfully matched with the node link.

[0094] In some embodiments, when the i-th object to be matched does not match the i-th node, compare the i-th object to be matched with the node pointed to by the failure pointer associated with the i-th node.

[0095] Specifically, when the i-th object to be matched does not match the i-th node, it means that the sequence to be matched does not match the current node link, and it is necessary to continue traversing other node links based on the sequence to be matched. Therefore, continue to compare the i-th object to be matched with the node pointed to by the failure pointer associated with the i-th node to traverse other multiple node links in the prefix tree.

[0096] In some embodiments, when the i-th object to be matched matches the i-th node and the i-th node is a leaf node, compare the (i + 1)-th object to be matched with the node pointed to by the failure pointer associated with the i-th node.

[0097] Specifically, when the i-th object to be matched matches the i-th node and the i-th node is a leaf node, it means that the traversal of the current node link has ended. Therefore, it is necessary to continue to compare the object to be matched next to the i-th object to be matched (i.e., the (i + 1)-th object to be matched) with the node pointed to by the failure pointer associated with the i-th node.

[0098] For example, referring to Figure 4 , traversing the prefix tree based on the sequence 401 to be matched. The prefix tree includes node links 301 and 302. The sequence 401 to be matched includes objects 4011 - 40112 to be matched. Node link 301 includes nodes 3011 - 30112, and node link 302 includes nodes 3021 - 3016. The index of the sequence 401 to be matched points to the object 4017 currently being matched. When the object 4011 to be matched successfully matches with node 3011, and node 3011 is a non - leaf node, then the object 4012 to be matched is continued to be matched with node 3012 (i.e., the child node of node 3011). The objects 4012 - 4016 to be matched are successively matched with nodes 3012 - 3016. Until when the index points to the object 4017, it is found that the object 4017 does not match with node 3017. Then, in the next step, the object 4017 is continued to be matched with node 3021 pointed to by the failure pointer of node 3017. The matching process of objects 4018 - 40112 to be matched is similar to the above steps and will not be elaborated here.

[0099] Step 203: When the matching results of multiple word segments meet the preset abnormal judgment conditions, determine the text to be recognized as abnormal text.

[0100] In some embodiments, at least one word segment whose matching result is hitting a node link is screened out from multiple word segments; when the number of at least one word segment is greater than a preset threshold, determine the text to be recognized as abnormal text.

[0101] Specifically, in the text to be recognized, the matching results of at least one word segment are hitting a node link, and the number of at least one word segment is greater than a preset threshold. The at least one word segment hitting the node link can be completely different or can include the same word segments (these same word segments hit the same node link). When the number of word segments hitting the node link is greater than the preset threshold, when the number of preset key texts appearing in the text to be recognized exceeds the preset threshold, the text to be recognized is considered as abnormal text.

[0102] In addition, based on the index of the sequence to be matched, record the position of the object to be matched in the sequence to be matched when the node link in the prefix tree is hit, and finally output the position where the preset key text corresponding to the node link appears in the text to be recognized.

[0103] For example, set the preset threshold to 100, create a prefix tree based on 10 preset key texts, and there are 1000 word segments in the text to be recognized. Among them, 120 word segments hit the node links in the prefix tree, that is, the number of word segments exceeds the preset threshold. These 120 word segments can be completely different or can contain some identical word segments. Additionally, it can be that all 10 node links have been hit, or some of the 10 node links have been hit, then the text to be recognized is considered an abnormal text, that is, the web page corresponding to the text to be recognized is a web page suspected of being involved in black and gray industries.

[0104] In the embodiments of the present application, the node links in the prefix tree are used to store the preset key texts and multiple reference bytes obtained by encoding the key texts. Compared with directly storing Chinese characters in nodes, it reduces the possibility of the prefix tree experiencing width explosion, which is equivalent to compressing the width of the prefix tree, thereby improving the efficiency and stability of abnormal text recognition. Secondly, for each word segment of the text to be recognized, the word segment and the corresponding multiple target bytes are used to sequentially traverse multiple node links in the prefix tree to obtain a matching result, and when the matching results of multiple word segments meet the preset abnormal judgment conditions, it is determined that the text to be recognized is an abnormal text. In this way, the final recognition result of the text to be recognized is determined through the matching results of multiple word segments, improving the accuracy of abnormal text recognition.

[0105] Based on the same technical concept, the embodiments of the present application provide a structural schematic diagram of an abnormal text recognition device, as Figure 5 shown. The abnormal text recognition device 500 includes:

[0106] An acquisition module 501, configured to acquire the text to be recognized and a preset prefix tree. The prefix tree includes: node links corresponding to multiple preset key texts, and each node link is used to store a preset key text and multiple reference bytes obtained by encoding the one preset key text.

[0107] A processing module 502, configured to perform the following operations respectively for multiple word segments in the text to be recognized: encode a word segment to obtain multiple target bytes; based on the one word segment and the multiple target bytes, sequentially traverse multiple node links in the prefix tree to obtain a matching result.

[0108] A judgment module 503, configured to determine that the text to be recognized is an abnormal text when the matching results of the multiple word segments meet the preset abnormal judgment conditions.

[0109] Optionally, the one preset key text includes multiple characters, and each character corresponds to multiple reference bytes;

[0110] The node link corresponding to the preset key text includes multiple nodes, and each node stores a character or a reference byte.

[0111] Optionally, each node in the node link corresponds to a failure pointer, and the failure pointer is used to indicate the next traversed node when the matching fails;

[0112] Among them, the failure pointer of the root node of the prefix tree points to the root node;

[0113] For each non-root node in the prefix tree, the failure pointer of the non-root node is determined based on the failure pointer of the parent node of the non-root node.

[0114] Optionally, the failure pointer of the non-root node is determined in the following manner:

[0115] Obtain the target node pointed to by the failure pointer of the parent node of the non-root node;

[0116] If at least one child node of the target node contains the same child node as the non-root node, set the failure pointer of the non-root node to point to the same child node;

[0117] If at least one child node of the target node does not contain the same child node as the non-root node, set the failure pointer of the non-root node to the root node.

[0118] Optionally, the processing module 502 is specifically configured to: split the word segmentation into multiple characters, and construct a sequence to be matched based on the multiple characters and the multiple target bytes;

[0119] Traverse the multiple node links in sequence based on the sequence to be matched;

[0120] For each node link, when there is a node set in the node link that matches the sequence to be matched, regard the sequence to be matched hitting the node link as the matching result.

[0121] Optionally, the processing module 502 is specifically configured to: compare the multiple objects to be matched in the sequence to be matched with the nodes at the corresponding positions in the node link in sequence, and when the comparison result meets the preset condition, regard the sequence to be matched hitting the node link as the matching result; the preset condition is that the comparison result of each object to be matched is a successful match;

[0122] The comparison process corresponding to each object to be matched includes the following operations:

[0123] Compare the i-th object to be matched in the sequence to be matched with the i-th node in the node link, where i is greater than 0;

[0124] When the i-th object to be matched matches the i-th node and the i-th node is a non-leaf node, compare the (i + 1)-th object to be matched in the sequence to be matched with the (i + 1)-th node in the node link.

[0125] Optionally, the processing module 502 is further configured to: when the i-th object to be matched does not match the i-th node, compare the i-th object to be matched with the node pointed to by the failure pointer associated with the i-th node.

[0126] Optionally, the processing module 502 is further configured to: when the i-th object to be matched matches the i-th node and the i-th node is a leaf node, compare the (i + 1)-th object to be matched with the node pointed to by the failure pointer associated with the i-th node.

[0127] Optionally, the determining module 503 is specifically configured to: screen out at least one word segment from the multiple word segments, where the matching result of the word segment hits the node link;

[0128] When the number of the at least one word segment is greater than a preset threshold, determine that the text to be recognized is an abnormal text.

[0129] In the embodiments of the present application, the node link in the prefix tree is used to store the preset key text and multiple reference bytes obtained by encoding the key text. Compared with directly storing Chinese characters in the node, it reduces the possibility of width explosion of the prefix tree, which is equivalent to compressing the width of the prefix tree, thereby improving the efficiency and stability of abnormal text recognition. Secondly, for each word segment of the text to be recognized, the word segment and the corresponding multiple target bytes are sequentially traversed through multiple node links in the prefix tree to obtain a matching result, and when the matching results of multiple word segments meet the preset abnormal judgment conditions, it is determined that the text to be recognized is an abnormal text. In this way, the final recognition result of the text to be recognized is determined through the matching results of multiple word segments, improving the accuracy of abnormal text recognition.

[0130] Based on the same technical concept, the embodiments of the present application provide a computer device, which may be Figure 1 the server shown in Figure 6 shown, including at least one processor 601 and a memory 602 connected to at least one processor. In the embodiments of the present application, the specific link medium between the processor 601 and the memory 602 is not limited. Figure 6 Taking the example that the processor 601 and the memory 602 are connected through a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0131] In an embodiment of the present application, the memory 602 stores instructions executed by at least one processor 601. By executing the instructions stored in the memory 602, the at least one processor 601 can perform the steps of the above-mentioned abnormal text recognition method.

[0132] Among them, the processor 601 is the control center of the computer device. It can connect various parts of the computer device through various interfaces and lines. By running or executing the instructions stored in the memory 602 and calling the data stored in the memory 602, the abnormal text recognition can be realized. Optionally, the processor 601 may include one or more processing modules. The processor 601 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 can be implemented on the same chip. In some embodiments, they can also be separately implemented on independent chips.

[0133] The processor 601 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, which can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0134] The memory 602 serves as a non-volatile computer-readable storage medium and can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The memory 602 may include at least one type of storage medium. For example, it may include flash memory, hard disks, multimedia cards, card-type memories, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memories, magnetic disks, optical disks, and so on. The memory 602 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer device, but is not limited thereto. The memory 602 in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function, for storing program instructions and / or data.

[0135] Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium that stores a computer program executable by a computer device. When the program runs on the computer device, it causes the computer device to execute the steps of the above-mentioned abnormal text recognition method.

[0136] Based on the same inventive concept, embodiments of the present application provide a computer program product, including a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to execute the steps of the above-mentioned abnormal text recognition method.

[0137] Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system, or a computer program product. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0138] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to the application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0139] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0141] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalent technologies, this application is also intended to include these modifications and variations.

Claims

1. An abnormal text recognition method, characterized in that, including: obtaining the text to be recognized and a preset prefix tree, where the prefix tree includes: node links corresponding to multiple preset key texts, each node link is used to store a preset key text, and multiple reference bytes obtained by encoding the one preset key text; performing the following operations respectively for multiple word segments in the text to be recognized: encoding a word segment to obtain multiple target bytes; based on the one word segment and the multiple target bytes, sequentially traversing multiple node links in the prefix tree to obtain a matching result; when the matching results of the multiple word segments meet a preset abnormal judgment condition, determining that the text to be recognized is an abnormal text.

2. The method according to claim 1, wherein the one preset key text includes multiple characters, and each character corresponds to multiple reference bytes; the node link corresponding to the one preset key text includes multiple nodes, and each node stores a character or a reference byte.

3. The method according to claim 1, characterized in that, each node in the node link corresponds to a failure pointer, and the failure pointer is used to indicate the next node to be traversed when the matching fails; wherein, the failure pointer of the root node of the prefix tree points to the root node; for each non-root node in the prefix tree, the failure pointer of the non-root node is determined based on the failure pointer of the parent node of the non-root node.

4. The method according to claim 3, characterized in that, the failure pointer of the non-root node is determined in the following manner: obtaining the target node pointed to by the failure pointer of the parent node of the non-root node; if at least one child node of the target node contains a child node identical to the non-root node, setting the failure pointer of the non-root node to point to the identical child node; if at least one child node of the target node does not contain a child node identical to the non-root node, setting the failure pointer of the non-root node to the root node.

5. The method according to any one of claims 1 to 4, characterized in that the sequentially traversing multiple node links in the prefix tree based on the one word segment and the multiple target bytes to obtain a matching result includes: splitting the one word segment into multiple characters, and constructing a sequence to be matched based on the multiple characters and the multiple target bytes; sequentially traversing the multiple node links based on the sequence to be matched; for each node link, when there is a node set in the node link that matches the sequence to be matched, hitting the sequence to be matched to the node link as the matching result.

6. The method according to claim 5, wherein the hitting the sequence to be matched to the node link as the matching result when there is a node set in the node link that matches the sequence to be matched includes: sequentially comparing multiple objects to be matched in the sequence to be matched with nodes at corresponding positions in the node link, and when the comparison result meets a preset condition, hitting the sequence to be matched to the node link as the matching result; the preset condition is: the comparison result of each object to be matched is a successful match; the comparison process corresponding to each object to be matched includes the following operations: comparing the i-th object to be matched in the sequence to be matched with the i-th node in the node link, where i is greater than 0; When the i-th object to be matched matches the i-th node and the i-th node is a non-leaf node, compare the (i + 1)-th object to be matched in the sequence to be matched with the (i + 1)-th node in the node link.

7. The method according to claim 6, wherein Further included: When the i-th object to be matched does not match the i-th node, compare the i-th object to be matched with the node pointed to by the failure pointer associated with the i-th node.

8. The method according to claim 6, wherein Further included: When the i-th object to be matched matches the i-th node and the i-th node is a leaf node, compare the (i + 1)-th object to be matched with the node pointed to by the failure pointer associated with the i-th node.

9. The method according to claim 5, characterized in that, When the matching results of the multiple word segmentations meet the preset abnormal judgment conditions, determining that the text to be recognized is an abnormal text includes: Filter out at least one word segmentation whose matching result is to hit the node link from the multiple word segmentations; When the number of the at least one word segmentation is greater than a preset threshold, determine that the text to be recognized is an abnormal text.

10. An apparatus for recognizing abnormal text, characterized in that, Including: An acquisition module for acquiring a text to be recognized and a preset prefix tree, where the prefix tree includes: node links corresponding to multiple preset key texts respectively, each node link is used to store a preset key text, and multiple reference bytes obtained by encoding the one preset key text; A processing module for respectively performing the following operations on multiple word segmentations in the text to be recognized: encoding one word segmentation to obtain multiple target bytes; based on the one word segmentation and the multiple target bytes, sequentially traverse multiple node links in the prefix tree to obtain a matching result; A judgment module for determining that the text to be recognized is an abnormal text when the matching results of the multiple word segmentations meet the preset abnormal judgment conditions.

11. A computer device, characterized in that, Including: A memory for storing program instructions; A processor for calling the program instructions stored in the memory and executing the steps of the method according to any one of claims 1 to 9 according to the obtained program.

12. A computer-readable storage medium, characterized in that, It stores a computer program executable by a computer device, and when the program runs on the computer device, the computer device executes the steps of the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program stored on a computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the method according to any one of claims 1 to 9.

Images