Method for constructing target word detection model, target word detection method and corresponding device
Patent Information
- Application Number
- CN202211704843.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-29
- Publication Date
- 2026-08-21
- Estimated Expiration
- 2042-12-29
AI Technical Summary
但是,基于这种方式无法准确检测出利用同音字等方式构建的敏感词,从而可能漏检部分敏感词,而且,还容易将一些包含敏感字符的正常词汇误判为敏感词,导致检测结果的准确性较差
[0011]The embodiments provided in this application firstly acquire multiple target words and multiple exclusion words, and establish a preset word library based on the multiple target words and multiple exclusion words. The target words include one or more target characters, and the exclusion words are non-target words formed by concatenating a first string composed of one or more target characters and a second string composed of one or more non-target characters. Secondly, based on the preset word library, a target word tree structure is constructed, wherein each branch structure of the target word tree structure represents a target word or an exclusion word, and each node in the branch structure corresponds to a character in the target word or exclusion word. Then, node attribute information is configured for each node in the target word tree structure to obtain a target word detection model, wherein the node attribute information is used to characterize the attribute information of the character corresponding to the current node and the attribute information of the characters corresponding to the child nodes of the current node. In this embodiment, since the preset vocabulary for constructing the target word detection model includes not only target words but also excluded words, and each node is configured with corresponding node attribute information, when performing detection based on this target word detection model, on the one hand, the target words constructed using homophones, near-homophones, and similar-looking characters can be accurately detected through the node attribute information; on the other hand, the node attribute information can also be used to reduce the situation of misdetecting normal words as target words, thereby further improving the accuracy of the target word detection results.
Smart Images

Figure CN116205215B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method for constructing a target word detection model, a target word detection method, and a corresponding device. Background Technology
[0002] With the continuous development of internet technology, people can access a vast amount of information online. However, some criminals take advantage of the convenience of online dissemination to publish illegal information, causing numerous negative impacts. To address this issue, technologies such as sensitive word detection can be used to review online information and promptly identify inappropriate content.
[0003] In related technologies, a sensitive word dictionary is typically built, and sensitive word detection is performed on the text to be processed based on this dictionary. However, this method cannot accurately detect sensitive words constructed using homophones or other methods, which may result in some sensitive words being missed. Moreover, it is easy to misclassify normal words containing sensitive characters as sensitive words, leading to poor accuracy of the detection results. Summary of the Invention
[0004] This application provides a method for constructing a target word detection model, a target word detection method, and corresponding devices, electronic devices, and storage media, which can detect target words more comprehensively and accurately, and reduce the situation of misdetecting normal words as target words.
[0005] In a first aspect, this application provides a method for constructing a target word detection model. The method includes: acquiring multiple target words and multiple exclusion words, and establishing a preset vocabulary based on the multiple target words and the multiple exclusion words. The target words include one or more target characters, and the exclusion words are non-target words formed by sequentially concatenating a first string composed of one or more target characters and a second string composed of one or more non-target characters. Based on the preset vocabulary, a target word tree structure is constructed, wherein each branch of the target word tree structure represents a target word or an exclusion word, and each node in the branch structure corresponds to a character in the target word or exclusion word. Node attribute information is configured for each node in the target word tree structure to obtain a target word detection model, wherein the node attribute information is used to characterize the attribute information of the character corresponding to the current node and the attribute information of the characters corresponding to the child nodes of the current node.
[0006] Secondly, this application provides a target word detection method, which includes: acquiring a text to be detected, the text including multiple characters; for the i-th character in the text to be detected, performing a first specified step, the first specified step including: determining whether there exists a first-level node matching the i-th character in a preset target word detection model, the first-level node being a child node of the root node in the target word detection model; if there exists a first-level node matching the i-th character in the target word detection model, cyclically performing a second specified step starting from the (i+1)-th character, the second specified step including: determining whether the j-th character matches a child node of a node matching the (j-1)-th character; if a match is found, determining whether a preset condition is met based on the exclusion attribute information corresponding to the j-th character; if the preset condition is not met... Under certain conditions, j is updated to j+1; wherein, the preset conditions include: the j-th character is the end of the target word, and the child node of the j-th character does not contain any non-target characters, or, the j-th character is the end of the exclusion word, wherein the end of the target word refers to the target character at the end of the target word, and the end of the exclusion word refers to the non-target character at the end of the exclusion word, i≥1, j≥i+1; until it is determined that the preset conditions are met according to the exclusion attribute information corresponding to the j-th character, the detection results of the i-th to j-th characters are obtained, and the first specified step is performed for the j+1-th character; after completing the detection of all characters in the text to be detected that are within the preset detection range, the target word detection result is obtained; wherein, the target word detection model is obtained by the construction method of the target word detection model described in any one of the embodiments of this application.
[0007] Thirdly, this application provides a device for constructing a target word detection model, which includes: a lexicon building module, used to acquire multiple target words and multiple exclusion words, and build a preset lexicon based on the multiple target words and multiple exclusion words, wherein the target words include one or more target characters, and the exclusion words are non-target words formed by sequentially concatenating a first string composed of one or more target characters and a second string composed of one or more non-target characters; a structure building module, used to construct a target word tree structure based on the preset lexicon, wherein each branch structure of the target word tree structure represents a target word or an exclusion word, and each node in the branch structure corresponds to a character in the target word or exclusion word; and an attribute configuration module, used to configure node attribute information for each node in the target word tree structure to obtain a target word detection model, wherein the node attribute information is used to characterize the attribute information of the character corresponding to the current node and the attribute information of the characters corresponding to the child nodes of the current node.
[0008] Fourthly, this application provides a target word detection device, which includes: an acquisition module for acquiring text to be detected, the text including multiple characters; a first execution module for executing a first specified step for the i-th character in the text to be detected, the first specified step including: determining whether there is a first-level node matching the i-th character in a preset target word detection model, the first-level node being a child node of the root node of the target word detection model; and a second execution module for cyclically executing a second specified step starting from the (i+1)-th character if a first-level node matching the i-th character exists in the target word detection model, the second specified step including: determining whether the j-th character matches a child node of a node matching the (j-1)-th character, and if a match is found, determining whether a preset condition is met based on the exclusion attribute information corresponding to the j-th character, and if not... Under the aforementioned preset conditions, j is updated to j+1; wherein, the preset conditions include: the j-th character is the end of a target word, and the child nodes of the j-th character do not contain non-target characters, or, the j-th character is the end of an exclusion word, wherein the end of a target word refers to the target character at the end of the target word, and the end of an exclusion word refers to the non-target character at the end of the exclusion word, i≥1, j≥i+1; the result determination module is used to obtain the detection results of the i-th to j-th characters until the exclusion attribute information corresponding to the j-th character satisfies the preset conditions, and to execute the first specified step for the j+1-th character; the total result acquisition module is used to obtain the target word detection result after completing the detection of all characters in the text to be detected that are within the preset detection range; wherein, the target word detection model is obtained by the construction method of the target word detection model described in any one of the embodiments of this application.
[0009] Fifthly, this application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to execute the above-described method for constructing a target word detection model or a target word detection method.
[0010] Fourthly, this application provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the above-described method for constructing a target word detection model or a target word detection method.
[0011] The embodiments provided in this application firstly acquire multiple target words and multiple exclusion words, and establish a preset word library based on the multiple target words and multiple exclusion words. The target words include one or more target characters, and the exclusion words are non-target words formed by concatenating a first string composed of one or more target characters and a second string composed of one or more non-target characters. Secondly, based on the preset word library, a target word tree structure is constructed, wherein each branch structure of the target word tree structure represents a target word or an exclusion word, and each node in the branch structure corresponds to a character in the target word or exclusion word. Then, node attribute information is configured for each node in the target word tree structure to obtain a target word detection model, wherein the node attribute information is used to characterize the attribute information of the character corresponding to the current node and the attribute information of the characters corresponding to the child nodes of the current node. In this embodiment, since the preset vocabulary for constructing the target word detection model includes not only target words but also excluded words, and each node is configured with corresponding node attribute information, when performing detection based on this target word detection model, on the one hand, the target words constructed using homophones, near-homophones, and similar-looking characters can be accurately detected through the node attribute information; on the other hand, the node attribute information can also be used to reduce the situation of misdetecting normal words as target words, thereby further improving the accuracy of the target word detection results.
[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description
[0013] The accompanying drawings are provided to further illustrate the present application and form part of the specification. They are used together with the embodiments of the present application to explain the application and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed example embodiments described with reference to the accompanying drawings, in which:
[0014] Figure 1 A flowchart illustrating a method for constructing a target word detection model provided in an embodiment of this application;
[0015] Figure 2 A schematic diagram of a target word detection model provided in an embodiment of this application;
[0016] Figure 3 A flowchart of a target word detection method provided in an embodiment of this application;
[0017] Figure 4 A flowchart of a target word detection method provided in an embodiment of this application;
[0018] Figure 5 A flowchart of a target word detection method provided in an embodiment of this application;
[0019] Figure 6 A block diagram of a device for constructing a target word detection model provided in an embodiment of this application;
[0020] Figure 7 A block diagram of a target word detection device provided in an embodiment of this application;
[0021] Figure 8 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0022] To enable those skilled in the art to better understand the technical solutions of this application, exemplary embodiments of this application are described below in conjunction with the accompanying drawings, including various details of the embodiments of this application to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0023] Where there is no conflict, the various embodiments of this application and the features thereof may be combined with each other.
[0024] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.
[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.
[0026] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.
[0027] In related technologies, a sensitive word dictionary is typically established, and sensitive word detection is performed on the text to be processed based on this dictionary. However, in practical applications, it is also necessary to exclude some normal words to avoid misclassifying them as target words, or to skip some words when detecting target words, or to filter out some special symbols when detecting target words.
[0028] In view of this, embodiments of this application provide a method for constructing a target word detection model, a target word detection method, and a corresponding apparatus. The target word detection model is constructed based on the target word and excluded words. When using the target word detection model for detection, not only can the target word be detected, but also the excluded words can be avoided as target words, thereby improving the accuracy of the detection results. In addition, by assigning rich node attribute information to each node in the target word detection model, the target word can be detected more comprehensively.
[0029] In summary, according to the target word detection model construction method of the embodiments of this application, since the preset vocabulary for constructing the target word detection model includes not only target words but also excluded words, and each node is configured with corresponding node attribute information, when performing detection based on the target word detection model, on the one hand, the target words constructed using homophones, near-homophones, and similar-looking characters can be accurately detected through the node attribute information; on the other hand, the node attribute information can also be used to reduce the situation of misdetecting normal words as target words, thereby further improving the accuracy of the target word detection results.
[0030] The target word detection model construction method and target word detection method according to the embodiments of this application can be executed by electronic devices such as terminal devices or servers. The terminal device can be user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. The server can be an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.
[0031] Firstly, embodiments of this application provide a method for constructing a target word detection model.
[0032] Figure 1 This is a flowchart illustrating a method for constructing a target word detection model according to an embodiment of this application. (Refer to...) Figure 1 The method includes:
[0033] In step S11, multiple target words and multiple exclusion words are obtained, and a preset word library is established based on the multiple target words and multiple exclusion words.
[0034] In some optional implementation manners, the target word includes one or more target characters, and the exclusion word is a non-target word formed by sequentially splicing a first string composed of one or more target characters and a second string composed of one or more non-target characters.
[0035] In some optional implementation manners, the target word can be a sensitive word, and the sensitive word refers to a word with a violent tendency, an unhealthy color, or an uncivilized word. The target word can also be some words specifically set according to specific requirements, and the embodiments of the present application do not limit this.
[0036] For example, the target word "尼玛" is an uncivilized Internet term, and "尼" and "玛" are the target characters that make up the target word.
[0037] In some optional implementation manners, there are some words formed by sequentially splicing a first string composed of one or more target characters and a second string composed of one or more non-target characters, that is, the exclusion word. For the exclusion word, because there are target characters, it is easy to be misjudged as a target word, resulting in inaccurate target word detection results.
[0038] For example, a first string composed of the target characters "尼" and "玛", and a second string composed of the non-target character "县" are sequentially spliced to form an exclusion word "尼玛县", and "尼玛县" is a place name and belongs to a non-target word. However, because it contains "尼玛", it is easy to be misjudged as a target word.
[0039] In step S12, a target word tree structure is constructed based on the preset word library.
[0040] In some optional implementation manners, the target word tree structure has a tree structure, and each branch structure of the target word tree structure represents a target word or an exclusion word, and each node in the branch structure corresponds to a character in the target word or the exclusion word.
[0041] In other words, each branch in the target word tree structure corresponds to a target word or an exclusion word, and the branch includes multiple nodes, and these nodes are sequentially serially connected by multiple characters that make up the target word or the exclusion word.
[0042] It should be noted that in the related art, the target word tree structure is a tree structure constructed based on the target word, so it can only be used to detect the target word. In the embodiments of the present application, the construction of the target word tree structure not only depends on the target word, but also includes the exclusion word. Therefore, not only can the target word be detected, but the exclusion word can also be prevented from being misjudged as the target word, thereby improving the accuracy of the detection result.
[0043] In step S13, node attribute information is configured for each node in the target word tree structure to obtain the target word detection model.
[0044] In some optional implementations, node attribute information is used to represent the attribute information of the character corresponding to the current node, as well as the attribute information of the characters corresponding to the child nodes of the current node. Here, a child node represents the next-level node of the current node, which is the node following the current node in the direction from the current node to the leaf node. A leaf node is the terminal node of the target word tree structure.
[0045] In other words, the essence of a target word detection model is a target word detection tree. The corresponding target word tree structure consists of target words and excluded words, and each node in the target word tree structure is assigned corresponding node attribute information, so that target words can be detected in the text to be processed based on the target word detection tree.
[0046] In some optional implementations, node attribute information includes at least one of the following: pronunciation attribute information, shape attribute information, exclusion attribute information, and wildcard information; wherein, pronunciation attribute information is used to characterize the pronunciation of the character corresponding to the current node, shape attribute information includes one or more similar-looking characters to the character corresponding to the current node, exclusion attribute information is used to determine whether the word corresponding to the current character belongs to the excluded word, and wildcard information is information about wildcards, which are pre-specified symbols used to represent any character (in other words, wildcards match any character).
[0047] For example, pronunciation attribute information can be represented by the pinyin, phonetic symbols, or other methods of characters.
[0048] In some optional implementations, the exclusion attribute information includes at least one of the following: word ending attribute information, exclusion word existence attribute information, and exclusion word attribute information; wherein, the word ending attribute information is used to characterize whether the character corresponding to the current node is a word ending character, the exclusion word existence attribute information is used to characterize whether there is a non-target character among the characters corresponding to one or more child nodes of the current node, and the exclusion word attribute information is used to characterize whether the character corresponding to the current node is a non-target character.
[0049] In some optional implementation manners, the node attribute information may further include traditional Chinese character attribute information or simplified Chinese character attribute information, etc. Among them, the traditional Chinese character attribute information is applied when the current character is a simplified Chinese character. The traditional Chinese character attribute information includes the traditional Chinese character corresponding to the character of the current node. Through the traditional Chinese character attribute information, a target word disguised based on the traditional Chinese character can be detected; the simplified Chinese character attribute information is applied when the current character is a traditional Chinese character, and includes the simplified Chinese character corresponding to the character of the current node. Through the simplified Chinese character attribute information, a target word disguised based on the simplified Chinese character can be detected.
[0050] The following Figure 2 expands and describes the target word detection model of the embodiments of the present application.
[0051] Figure 2 is a schematic diagram of a target word detection model provided for the embodiments of the present application. Referring to Figure 2 , the target word detection model at least shows target words such as "cash embezzled**", "fund embezzlement", "Nima", and exclusion words such as "Nima Town" and "Nima County". Among them, the root node is used to connect each branch to form a tree structure. Moreover, usually, the node attribute information of the root node is empty.
[0052] As Figure 2 shown, one branch along the leftmost side of the root node corresponds to the target word "cash embezzled**", where "*" is a wildcard that matches any character and is used to expand the detection range of the target word. For example, based on "cash embezzled**", target words such as "cash embezzled from Zhang San" and "cash embezzled from Li Si" can be detected. For this target word, "tao" is the first character of the target word, and the node corresponding to it is directly connected to the root node. Therefore, the node corresponding to "tao" is a first-level node, the node corresponding to "qu" is a child node of this first-level node, the first "*" is a child node of the node corresponding to "qu", and so on. The node corresponding to "jin" is a child node of the node corresponding to "xian".
[0053] Furthermore, for the node corresponding to "tao", its node attribute information includes pinyin, similar-shaped characters, word-ending attribute, exclusion word attribute, and exclusion word existence attribute. Among them, the pinyin "tao" is the pinyin of "tao"; the similar-shaped characters are characters similar in shape to "tao", including "kua" and "she"; the word-ending attribute is false, which is used to indicate that "tao" is not the last character of this target word; the exclusion word attribute is false, which is used to indicate that "tao" is not a non-target character; the exclusion word existence attribute is false, which is used to indicate that the character corresponding to the child node of "tao" does not have a non-target character (that is, the character "qu" is not a non-target character). Other nodes are similar to this and will not be further described here.
[0054] For the target word "尼玛" and the exclusion words "尼玛镇" and "尼玛县", since they have the same target characters "尼" and "玛", a tree structure with a branch shape is formed. Hereinafter, the node attribute information of the nodes corresponding to "玛" and "镇" will be explained in detail. For the node corresponding to "玛", on the one hand, it is the last character of the target word "尼玛", and on the other hand, it is the target character in the exclusion word "尼玛镇". Therefore, in the node attribute information of this node, the pinyin "ma" is the pinyin of "玛"; the similar-shaped characters are the characters similar in shape to "玛", including "码", "吗", "马"; the end-of-word attribute is true, which is used to indicate that "玛" is the last character of the target word "尼玛"; the exclusion-word attribute is false, which is used to indicate that "玛" is not a non-target character (i.e., "玛" is a target character); the exclusion-word existence attribute is true, which is used to indicate that there are non-target characters in the characters corresponding to the child nodes of "玛" (i.e., there are non-target characters "镇" and "县" after the character "玛"). For the node corresponding to "镇", it is the last character of the exclusion word "尼玛镇". In the node attribute information of this node, the pinyin "zhen" is the pinyin of "镇"; the similar-shaped characters are in the default state; the end-of-word attribute is true, which is used to indicate that "镇" is the last character of the exclusion word "尼玛镇"; the exclusion-word attribute is true, which is used to indicate that "镇" is a non-target character; the exclusion-word existence attribute is false, which is used to indicate that there are no non-target characters in the characters corresponding to the child nodes of "镇" (i.e., the character "镇" is the last non-target character of the exclusion word).
[0055] Second, the embodiments of the present application provide a method for detecting target words.
[0056] Figure 3 is a flowchart of a method for detecting target words provided by the embodiments of the present application. Refer to Figure 3 and the method includes:
[0057] In step S31, the text to be detected is obtained, and the text to be detected includes multiple characters.
[0058] In some optional implementation manners, the text to be detected may be text obtained from forums, microblogs, comments, etc., or may be text obtained by other means, and the embodiments of the present application do not limit this.
[0059] In some optional implementation manners, a preset detection range may also be set for the text to be detected. Through the preset detection range, it can be determined which text content in the text to be detected is to be detected for target words.
[0060] In step S32, for the i-th character in the text to be detected, perform a first specified step, where the first specified step includes: determining whether there is a first-level node in the preset target word detection model that matches the i-th character, and the first-level node is a child node of the root node in the target word detection model.
[0061] Among them, the target word detection model is obtained by using the construction method of the target word detection model according to any embodiment of the present application.
[0062] In some optional implementation manners, the text to be detected is "I come from Nima Town. I came here this time to attend a meeting". For the first character "I", determine whether there is a first-level node in the target word detection model that matches "I". Among them, the first-level node is a child node of the root node in the target word detection model, that is, a node directly connected to the root node. By determining whether there is a first-level node in the target word detection model that matches the i-th character, it can be judged whether the i-th character belongs to the first character of a target word or an exclusion word. When the i-th character does not belong to the first character of a target word or an exclusion word, jump to the (i + 1)-th character, and determine whether there is a first-level node in the target word detection model that matches the (i + 1)-th character to start a new round of detection (for example, jump to the second character "come" and determine whether there is a first-level node that matches "come" through the target word detection model).
[0063] In some optional implementation manners, when the i-th character is a preset special character, perform the first specified step on the (i + 1)-th character. Based on this, the special character can be skipped, and the subsequent characters can be detected to avoid the influence of the special character on the target word detection.
[0064] Among them, a special character refers to a symbol that has a relatively low usage frequency and is usually difficult to directly input compared to traditional or common symbols (for example, Roman characters). In some optional implementation manners, in order to interfere with the target word detection, one or more special characters are set between the target characters.
[0065] It should be noted that a wildcard is a symbol used to represent any character, and its essence and connotation are different from those of special characters. Usually, a certain character or symbol can be specified in advance according to experience, statistical data, actual needs, etc. as a wildcard to avoid misjudging a wildcard as a special character or misjudging a special character as a wildcard during the detection process. For example, it is specified in advance that "*" is a wildcard, and during the detection process, it is considered that "*" matches any character.
[0066] In some optional implementations, step S32 includes: determining whether there is a target character that is the same as the i-th character among the target characters corresponding to each first-level node of the target word detection model; if there is a target character that is the same as the i-th character, determining that there is a first-level node that matches the i-th character.
[0067] It should be noted that matching the i-th character with a first-level node does not simply mean that the i-th character is identical to the corresponding character in the first-level node. To avoid missing target words constructed based on homophones, similar-looking characters, etc., the comprehensiveness of target word detection can be improved by utilizing the node attribute information of the first-level nodes.
[0068] In some optional implementations, each node in the target word detection model includes node attribute information; the node attribute information includes pronunciation attribute information and shape attribute information; correspondingly, if there is no target character identical to the i-th character, the pronunciation of the i-th character is determined; based on the pronunciation attribute information of each first-level node in the target word detection model, it is determined whether there is a first-level node whose pronunciation similarity to the i-th character is greater than a preset pronunciation threshold; if there is a first-level node whose pronunciation similarity to the i-th character is greater than the preset pronunciation threshold (e.g., two characters have the same pronunciation but different tones; for example, one character is a front nasal sound and the other is a back nasal sound; for example, one character is a retroflex sound and the other is a flat sound), it is determined whether there is a target character identical to the i-th character. The first-level node matching the i-th character is determined. If no first-level node exists that has a pronunciation similarity greater than a preset pronunciation threshold with the i-th character, the shape attribute information of each first-level node in the target word detection model is used to determine if a first-level node exists that has a glyph similarity greater than a preset glyph threshold with the i-th character. If a first-level node exists that has a glyph similarity greater than a preset glyph threshold with the i-th character, a first-level node matching the i-th character is determined. If no first-level node exists that has a glyph similarity greater than a preset glyph threshold with the i-th character, the target word detection model is determined if a first-level node in the target word detection model has a wildcard target character. If a first-level node exists that has a wildcard target character, a first-level node matching the i-th character is determined.
[0069] One approach is to pre-set multiple similar-looking characters in the shape attribute information, and determine whether the character similarity is less than the preset character shape threshold by judging whether the character to be detected belongs to one of the similar-looking characters. Alternatively, the shape features of the characters can be extracted using methods such as neural networks, and the character shape similarity between characters can be determined by comparing the shape features.
[0070] It should be noted that the above methods for determining glyph similarity are merely illustrative examples, and the embodiments of this application do not impose any limitations on them.
[0071] In step S33, if there is a first-level node in the target word detection model that matches the i-th character, the second specified step is executed cyclically starting from the (i+1)-th character. The second specified step includes: determining whether the j-th character matches the child node of the node that matches the (j-1)-th character; if they match, determining whether the preset conditions are met based on the exclusion attribute information corresponding to the j-th character; if the preset conditions are not met, updating j to j+1.
[0072] Where i ≥ 1, j ≥ i + 1. The preset conditions include: the j-th character ends in the target word, and none of the j-th character's child nodes are non-target characters; or, the j-th character ends in an excluded word. The target word ending refers to the target character at the end of the target word, and the excluded word ending refers to the non-target character at the end of the excluded word. In other words, for multiple characters after the i-th character, the second specified step is executed sequentially until the preset conditions are met.
[0073] When there is a first-level node in the target word detection model that matches the i-th character, it means that the i-th character belongs to the first character of the target word or the excluded word. Then, it is necessary to start from the (i+1)-th character and determine whether it matches the subsequent characters of the target word or the excluded word (i.e., the characters after the first character).
[0074] In some optional implementations, a second specified step is performed iteratively starting from the (i+1)th character to determine whether these characters match subsequent characters of the target word or excluded word. The second specified step includes: determining whether the j-th character matches a child node of the node that matches the (j-1)-th character; if a match is found, determining the exclusion attribute information corresponding to the j-th character. After obtaining the exclusion attribute information corresponding to the j-th character, it is determined whether a preset condition is met based on this information; if the preset condition is not met, j = j++ is executed, and the second specified step is performed on the new j-th character.
[0075] In some optional implementations, the exclusion attribute information includes word ending attribute information, exclusion word existence attribute information, and exclusion word attribute information. The step of determining whether the preset condition is met based on the exclusion attribute information corresponding to the j-th character includes: determining whether the j-th character is a word ending character based on the word ending attribute information of the node matching the j-th character; if the j-th character is a word ending character, determining whether the j-th character is a non-target character based on the exclusion word attribute information of the node matching the j-th character; if the j-th character is a non-target character, determining that the j-th character belongs to the exclusion word ending and determining that the preset condition is met; if the j-th character is not a non-target character, determining that the j-th character belongs to the target word ending and determining whether there are non-target characters in the child nodes of the j-th character based on the exclusion word existence attribute information of the j-th character; if there are no non-target characters in the child nodes of the j-th character, determining that the preset condition is met.
[0076] In some optional implementations, after determining whether there is a non-target character in the child node of the j-th character, the method further includes: if there is a non-target character in the child node of the j-th character, performing a second specified step on the (j+1)-th character.
[0077] In some optional implementations, after determining in step S33 whether the j-th character matches the child node of the node that matches the (j-1)-th character, the method further includes: if it is determined that the j-th character does not match the child node of the node that matches the (j-1)-th character, determining that the i-th to (j-1)-th characters are non-target words, and performing the first specified step for the j-th character.
[0078] In step S34, the detection results of characters i to j are obtained until the preset conditions are met based on the exclusion attribute information corresponding to the j-th character, and the first specified step is executed for the (j+1)-th character.
[0079] In some optional implementations, if the preset condition is that the j-th character ends with the target word and the child node of the j-th character does not contain a non-target character, the detection results of the i-th to j-th characters are determined as the target word; if the preset condition is that the j-th character ends with an excluded word, the detection results of the i-th to j-th characters are determined as the excluded word.
[0080] The text to be detected is "I come from Nima Town, and I'm here to attend a meeting" and Figure 2Taking the target word detection model shown as an example for illustration. For the 1st character "我", there is no matching first-level node in the target word detection model, so it jumps to the 2nd character "来" for detection; since there is no first-level node matching "来" in the target word detection model, it jumps to the 3rd character "自" for detection; since there is no first-level node matching "自" in the target word detection model, it jumps to the 4th character "尼" for detection; since there is a first-level node matching "尼" in the target word detection model, it is necessary to determine whether the 5th character "玛" matches the child node of the node corresponding to "尼"; since the 5th character "玛" matches the child node of the node corresponding to "尼", and according to the exclusion attribute information of the node corresponding to "玛", it is determined that "玛" is the end of the target word, and there are non-target characters in its child nodes, so it is determined that the preset condition is not satisfied, and it is necessary to determine whether the 6th character "镇" matches the child node of the node corresponding to "玛"; since the 6th character "镇" matches the child node of the node corresponding to "玛", and further according to the exclusion attribute information of the node corresponding to "镇", it is determined that "镇" is the end of the exclusion word, so it is determined that the preset condition is satisfied, and "尼玛镇" is determined as the exclusion word; then, starting from the 7th character "这", a new target word detection is carried out, and the detection process is similar to the above content and will not be elaborated here.
[0081] In step S35, when the detection of all characters within the preset detection range in the text to be detected is completed, the target word detection result is obtained.
[0082] In this embodiment, firstly, the text to be detected is obtained, and the target word detection object is identified; secondly, for the i-th character in the text to be detected, a first specified step is executed. The first specified step includes: determining whether there exists a first-level node matching the i-th character in the preset target word detection model. The first-level node is a child node of the root node in the target word detection model. Through this step, it can be determined whether the i-th character is the first character of the target word or the excluded word, thereby clarifying what judgment operation should be performed on the (i+1)-th character; furthermore, if there exists a first-level node matching the i-th character in the target word detection model, a second specified step is executed cyclically starting from the (i+1)-th character. The second specified step includes: determining whether the j-th character matches the child node of the node matching the (j-1)-th character. In the case of a match, the system determines whether a preset condition is met based on the exclusion attribute information corresponding to the j-th character. If the preset condition is not met, j is updated to j+1. The preset conditions include: the j-th character ends with the target word, and the child nodes of the j-th character do not contain non-target characters; or, the j-th character ends with the exclusion word, where the target word ending refers to the target character at the end of the target word, and the exclusion word ending refers to the non-target character at the end of the exclusion word, i≥1, j≥i+1. The detection results for characters i to j are obtained until the preset condition is met based on the exclusion attribute information corresponding to the j-th character. Then, the first specified step is executed for the j+1-th character. After detecting all characters within the preset detection range in the text to be detected, the target word detection result is obtained. In this embodiment, on the one hand, the node attribute information can accurately detect target words constructed using homophones, near-homophones, and similar-looking characters. On the other hand, the exclusion attribute information in the node attribute information can reduce the misdetection of normal words as target words, thereby further improving the accuracy of the target word detection results.
[0083] Figure 4 This is a flowchart illustrating a target word detection method provided in an embodiment of this application. (Refer to...) Figure 4 The method includes:
[0084] S401, Obtain the text to be detected.
[0085] S402, based on the preset detection range, determine the detection position and the total string length N, and set the position identifier i=1 corresponding to the first character to be detected.
[0086] S403: Starting from the character corresponding to position identifier i, target word detection is performed by matching it with characters in the target word detection model.
[0087] S404, obtain the hit end position S corresponding to this detection.
[0088] The "hit end position" indicates the string length of the target word when a character in the text to be detected matches the target word in the target word detection model.
[0089] For example, when S=0, it means that there is no matching target word. When S=3, it means that the string length of the target word matched in the text to be detected during this detection process is 3, that is, in the text to be detected, the i-th character to the (i+2)-th character matches the corresponding target word.
[0090] S405 determines whether the hit end position S is greater than zero.
[0091] If the hit end position S is equal to zero, proceed to step S406; if the hit end position S is greater than zero, proceed to step S407.
[0092] S406, Update the position identifier i = i++.
[0093] S407. Based on the position identifier i and the hit end position S, obtain the string consisting of [i, i+S-1] in the text to be detected corresponding to the target word, and jump to step S408.
[0094] S408, update the position identifier i to i+S.
[0095] The purpose of updating the position identifier in step S406 or S408 is to detect subsequent undetected characters. Furthermore, after executing either step S406 or S408, the process jumps to step S409.
[0096] S409, determine if i is less than the total length of the string.
[0097] If i is less than the total length of the string, proceed to step S403 to begin a new round of detection.
[0098] If i equals the total length of the string, it means that the detection of all characters in the preset detection range in the text to be detected has been completed, and the corresponding target word detection result has been obtained, so the process ends.
[0099] Figure 5 This is a flowchart illustrating a target word detection method provided in an embodiment of this application. (Refer to...) Figure 5 The method includes:
[0100] S501, Load the text to be detected, determine the start position of the detection, set the position identifier i=0 corresponding to the first character to be detected, and initialize the end position of the hit to 0.
[0101] S502, extract the character with position identifier i from the text to be detected as the character to be detected.
[0102] S503, determine whether the i-th character is a special character.
[0103] If the i-th character is a special character, proceed to step S504; if the i-th character is not a special character, proceed to step S505.
[0104] Step S504: Update the position identifier i = i++, and jump to step S502 to start a new detection process.
[0105] S505, determine whether the i-th character matches the first-level node in the target word detection model.
[0106] If the i-th character does not match the first-level node, proceed to step S506; if the i-th character matches the first-level node, proceed to step S507.
[0107] S506, return to the hit end position S=0, and jump to step S504.
[0108] S507, determine whether the j-th character matches the child node of the node corresponding to the (j-1)-th character.
[0109] If the j-th character matches a child node, proceed to step S508; otherwise, proceed to step S506.
[0110] Where j-1≥i, and when j-1=i is a special case, it is equivalent to judging whether the (i+1)th character matches the child node of the first-level node corresponding to the ith character.
[0111] Step S508: Determine whether the j-th character is the end character of a word.
[0112] If the j-th character is the end character of the word, proceed to step S509. If the j-th character is not the end character of the word, take the (j+1)-th character as the new j-th character (equivalent to j = j++) and proceed to step S507.
[0113] S509, determine whether the j-th character is a non-target character.
[0114] If the j-th character is not the target character, proceed to step S506; if the j-th character is the target character, proceed to step S510.
[0115] Step S510: Update the hit end position S according to the number of cycles in step S507.
[0116] For example, if step S507 loops once, it means that the i-th character and the (i+1)-th character (where j = i+1) match the characters in the target word detection model, and therefore, the hit end position S = 2 is determined; if S507 loops three times, it means that the i-th character to the (i+3)-th character (where j = i+3) match the characters in the target word detection model, and therefore, the hit end position S = 4 is determined.
[0117] Step S511: Determine whether there is a non-target character in the child node of the j-th character.
[0118] If a non-target character exists in the child node of the j-th character, the (j+1)-th character is taken as the new j-th character, and the process jumps to step S507; if no non-target character exists in the child node of the j-th character, the process jumps to step S512.
[0119] S512, return to the end position S of the hit, and determine the target word detected this time based on the end position S of the hit.
[0120] As described in step S510, when S=2, it means that the detected target word corresponds to the i-th character and the (i+1)-th character; when S=4, it means that the detected target word corresponds to the i-th character to the (i+3)-th character.
[0121] S513, update i = i + S - 1, and jump to step S502.
[0122] The target word detection process ends when all characters have been detected and all target words have been obtained.
[0123] Thirdly, embodiments of this application provide an apparatus for constructing a target word detection model.
[0124] Figure 6 This is a block diagram of a device for constructing a target word detection model according to an embodiment of this application.
[0125] Reference Figure 6 This application provides an apparatus for constructing a target word detection model. The apparatus 600 for constructing the target word detection model includes:
[0126] The lexicon building module 601 is used to acquire multiple target words and multiple excluded words, and to build a preset lexicon based on the multiple target words and multiple excluded words. The target words include one or more target characters, and the excluded words are non-target words formed by concatenating a first string composed of one or more target characters and a second string composed of one or more non-target characters.
[0127] The structure building module 602 is used to build a target word tree structure based on a preset lexicon. Each branch of the target word tree structure represents a target word or an excluded word, and each node in the branch structure corresponds to a character in the target word or an excluded word.
[0128] The attribute configuration module 603 is used to configure node attribute information for each node in the target word tree structure to obtain the target word detection model. The node attribute information is used to represent the attribute information of the character corresponding to the current node and the attribute information of the characters corresponding to the child nodes of the current node.
[0129] In this embodiment, firstly, multiple target words and multiple exclusion words are acquired, and a preset word library is established based on the multiple target words and multiple exclusion words. The target words include one or more target characters, and the exclusion words are non-target words formed by concatenating a first string composed of one or more target characters and a second string composed of one or more non-target characters. Secondly, based on the preset word library, a target word tree structure is constructed, wherein each branch of the target word tree structure represents a target word or an exclusion word, and each node in the branch structure corresponds to a character in the target word or exclusion word. Then, node attribute information is configured for each node in the target word tree structure to obtain a target word detection model, wherein the node attribute information is used to characterize the attribute information of the character corresponding to the current node and the attribute information of the characters corresponding to the child nodes of the current node. In this embodiment, since the preset vocabulary for constructing the target word detection model includes not only target words but also excluded words, and each node is configured with corresponding node attribute information, when performing detection based on this target word detection model, on the one hand, the target words constructed using homophones, near-homophones, and similar-looking characters can be accurately detected through the node attribute information; on the other hand, the node attribute information can also be used to reduce the situation of misdetecting normal words as target words, thereby further improving the accuracy of the target word detection results.
[0130] Fourthly, embodiments of this application provide a target word detection device.
[0131] Figure 7 This is a block diagram of a target word detection device provided in an embodiment of this application.
[0132] Reference Figure 7 This application provides a target word detection device 700, which includes:
[0133] The acquisition module 701 is used to acquire the text to be detected, which includes multiple characters.
[0134] The first execution module 702 is used to perform a first specified step for the i-th character in the text to be detected. The first specified step includes: determining whether there is a first-level node in the preset target word detection model that matches the i-th character. The first-level node is a child node of the root node of the target word detection model.
[0135] The second execution module 703 is used to cyclically execute a second specified step starting from the (i+1)th character when there is a first-level node matching the i-th character in the target word detection model. The second specified step includes: determining whether the j-th character matches the child node of the node matching the (j-1)-th character; if they match, determining whether a preset condition is met based on the exclusion attribute information corresponding to the j-th character; if the preset condition is not met, updating j to j+1. The preset conditions include: the j-th character is the end of the target word, and the child node of the j-th character does not contain any non-target characters, or the j-th character is the end of an exclusion word. The end of the target word refers to the target character at the end of the target word, and the end of the exclusion word refers to the non-target character at the end of the exclusion word, i≥1, j≥i+1.
[0136] The result determination module 704 is used to obtain the detection results of the i-th to j-th characters until the exclusion attribute information corresponding to the j-th character meets the preset conditions, and to perform the first specified step for the (j+1)-th character.
[0137] The total result acquisition module 705 is used to obtain the target word detection result after completing the detection of all characters in the text to be detected that are within the preset detection range;
[0138] The target word detection model is obtained using the construction method of the target word detection model in any one of the embodiments of this application.
[0139] In this embodiment, firstly, the text to be detected is obtained, and the target word detection object is identified; secondly, for the i-th character in the text to be detected, a first specified step is executed. The first specified step includes: determining whether there exists a first-level node matching the i-th character in the preset target word detection model. The first-level node is a child node of the root node in the target word detection model. Through this step, it can be determined whether the i-th character is the first character of the target word or the excluded word, thereby clarifying what judgment operation should be performed on the (i+1)-th character; furthermore, if there exists a first-level node matching the i-th character in the target word detection model, a second specified step is executed cyclically starting from the (i+1)-th character. The second specified step includes: determining whether the j-th character matches the child node of the node matching the (j-1)-th character. In the case of a match, the system determines whether a preset condition is met based on the exclusion attribute information corresponding to the j-th character. If the preset condition is not met, j is updated to j+1. The preset conditions include: the j-th character ends with the target word, and the child nodes of the j-th character do not contain non-target characters; or, the j-th character ends with the exclusion word, where the target word ending refers to the target character at the end of the target word, and the exclusion word ending refers to the non-target character at the end of the exclusion word, i≥1, j≥i+1. The detection results for characters i to j are obtained until the preset condition is met based on the exclusion attribute information corresponding to the j-th character. Then, the first specified step is executed for the j+1-th character. After detecting all characters within the preset detection range in the text to be detected, the target word detection result is obtained. In this embodiment, on the one hand, the node attribute information can accurately detect target words constructed using homophones, near-homophones, and similar-looking characters. On the other hand, the exclusion attribute information in the node attribute information can reduce the misdetection of normal words as target words, thereby further improving the accuracy of the target word detection results.
[0140] It is understood that the various method embodiments mentioned above in this application can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this application will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.
[0141] In addition, this application also provides electronic devices and computer-readable storage media, all of which can be used to implement any of the target word detection model construction methods or target word detection methods provided in this application. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.
[0142] Figure 8 This is a block diagram of an electronic device provided in an embodiment of this application.
[0143] Reference Figure 8 This application provides an electronic device 800, which includes: at least one processor 801; at least one memory 802; and one or more I / O interfaces 803 connected between the processor 801 and the memory 802; wherein the memory 802 stores one or more computer programs that can be executed by the at least one processor 801, and the one or more computer programs are executed by the at least one processor 801 to enable the at least one processor 801 to execute the above-described method for constructing a target word detection model or a target word detection method.
[0144] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the above-described method for constructing a target word detection model or a target word detection method. The computer-readable storage medium may be volatile or non-volatile.
[0145] This application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described method for constructing the target word detection model or the target word detection method.
[0146] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).
[0147] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0148] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0149] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.
[0150] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.
[0151] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0152] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0153] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0155] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.
Claims
1. A method for constructing a target word detection model, characterized in that, include: Multiple target words and multiple excluded words are obtained, and a preset word library is established based on the multiple target words and the multiple excluded words. The target words include one or more target characters, and the excluded words are non-target words formed by concatenating a first string composed of one or more target characters and a second string composed of one or more non-target characters. Based on the preset vocabulary, a target word tree structure is constructed, wherein each branch of the target word tree structure represents a target word or an excluded word, and each node in the branch structure corresponds to a character in the target word or the excluded word; Configure node attribute information for each node in the target word tree structure to obtain a target word detection model. The node attribute information is used to characterize the attribute information of the character corresponding to the current node and the attribute information of the characters corresponding to the child nodes of the current node. The node attribute information includes pronunciation attribute information, shape attribute information, wildcard information and exclusion attribute information. The target word detection model is configured as follows: during the detection process, if there is no node that exactly matches a character in the first-level node of the target word detection model, the model determines whether there is a matching first-level node in turn based on the pronunciation attribute information, shape attribute information and wildcard information; in subsequent matching, the model distinguishes between excluded words and target words based on the exclusion attribute information, and the target word detection result is obtained after traversal is completed.
2. The method according to claim 1, characterized in that, The pronunciation attribute information is used to characterize the pronunciation of the character corresponding to the current node; the shape attribute information includes one or more similar-looking characters to the character corresponding to the current node; the exclusion attribute information is used to determine whether the word corresponding to the current character is an excluded word; the wildcard information is information about wildcards, and the wildcards are pre-specified symbols used to represent any character.
3. The method according to claim 2, characterized in that, The exclusion attribute information includes at least one of word ending attribute information, exclusion word existence attribute information, and exclusion word attribute information; The word ending attribute information is used to characterize whether the character corresponding to the current node is a word ending character. The exclusion word existence attribute information is used to characterize whether there are non-target characters among the characters corresponding to one or more child nodes of the current node. The exclusion word attribute information is used to characterize whether the character corresponding to the current node is a non-target character.
4. A target word detection method, characterized in that, include: Obtain the text to be detected, which includes multiple characters; For the i-th character in the text to be detected, a first specified step is performed, the first specified step including: determining whether there is a first-level node in the preset target word detection model that matches the i-th character, the first-level node being a child node of the root node in the target word detection model; If a first-level node matching the i-th character exists in the target word detection model, the second specified step is executed cyclically starting from the (i+1)-th character. The second specified step includes: determining whether the j-th character matches the child node of the node matching the (j-1)-th character; if a match is found, determining whether a preset condition is met based on the exclusion attribute information corresponding to the j-th character; if the preset condition is not met, updating j to j+1. The preset condition includes: the j-th character is the end of the target word, and the child node of the j-th character does not contain any non-target characters, or the j-th character is the end of an exclusion word. The end of the target word refers to the target character at the end of the target word, and the end of the exclusion word refers to the non-target character at the end of the exclusion word, i≥1, j≥i+1. Until the preset condition is met based on the exclusion attribute information corresponding to the j-th character, the detection results of the i-th to j-th characters are obtained, and the first specified step is executed for the (j+1)-th character; After completing the detection of all characters in the text to be detected that are within the preset detection range, the target word detection result is obtained; wherein, the target word detection model is obtained by the construction method of the target word detection model as described in any one of claims 1-3.
5. The method according to claim 4, characterized in that, The step of determining whether a first-level node matching the i-th character exists in the preset target word detection model includes: Determine whether there exists a target character that is the same as the i-th character among the target characters corresponding to each first-level node of the target word detection model; If there exists a target character that is the same as the i-th character, then there exists a first-level node that matches the i-th character.
6. The method according to claim 5, characterized in that, Each node in the target word detection model includes node attribute information; the node attribute information includes pronunciation attribute information, shape attribute information, and wildcard information. After determining whether there exists a target character identical to the i-th character among the target characters corresponding to each first-level node of the target word detection model, the process further includes: Determine the pronunciation of the i-th character if no target character is identical to the i-th character. Based on the pronunciation attribute information of each first-level node of the target word detection model, determine whether there exists a first-level node whose pronunciation similarity to the i-th character is greater than a preset pronunciation threshold; If there exists a first-level node whose pronunciation similarity to the i-th character is greater than a preset pronunciation threshold, then it is determined that there exists a first-level node that matches the i-th character. In the absence of a first-level node whose pronunciation similarity to the i-th character is greater than a preset pronunciation threshold, the shape attribute information of each first-level node of the target word detection model is used to determine whether there is a first-level node whose shape similarity to the i-th character is greater than a preset shape threshold. If there is a first-level node whose glyph similarity to the i-th character is greater than the preset glyph threshold, then it is determined that there is a first-level node that matches the i-th character. In the absence of a first-level node whose glyph similarity to the i-th character is greater than a preset glyph threshold, determine whether there is a first-level node in the target word detection model where the target character is a wildcard. If there exists a first-level node where the target character is a wildcard, then there exists a first-level node that matches the i-th character.
7. The method according to claim 4, characterized in that, Before determining whether a first-level node matching the i-th character exists in the preset target word detection model, the method further includes: If the i-th character is a preset special character, perform the first specified step on the (i+1)-th character.
8. The method according to claim 4, characterized in that, The exclusion attribute information includes word ending attribute information, exclusion word existence attribute information, and exclusion word attribute information; The steps for determining whether the preset conditions are met based on the exclusion attribute information corresponding to the j-th character include: Based on the word ending attribute information of the node that matches the j-th character, determine whether the j-th character is the word ending character; If the j-th character is the end character of a word, determine whether the j-th character is a non-target character based on the exclusion word attribute information of the node that matches the j-th character; If the j-th character is not the target character, determine that the j-th character belongs to the end of the excluded word and determine that the preset condition is met; If the j-th character is not a non-target character, determine that the j-th character belongs to the end of the target word, and determine whether there are non-target characters in the child nodes of the j-th character based on the exclusion word existence attribute information of the j-th character. If there is no non-target character in the child node of the j-th character, the preset condition is satisfied.
9. The method according to claim 8, characterized in that, After determining whether the child node of the j-th character contains a non-target character, the process further includes: If a non-target character exists in the child node of the j-th character, perform the second specified step on the (j+1)-th character.
10. The method according to claim 4, characterized in that, After determining whether the j-th character matches the child node of the node that matches the (j-1)-th character, the process further includes: If the child nodes of the node that matches the j-th character do not match, the i-th to j-1 characters are determined to be non-target words, and the first specified step is performed for the j-th character.
11. The method according to claim 4, characterized in that, The process of obtaining the detection results of characters i to j until the preset condition is met based on the exclusion attribute information corresponding to the j-th character includes: Under the preset condition that the j-th character ends with the target word and that the child node of the j-th character does not contain any non-target characters, the detection results of the i-th to j-th characters are determined to be the target word; If the preset condition is that the j-th character ends with an excluded word, the detection results of the i-th to j-th characters are determined as excluded words.
12. A device for constructing a target word detection model, characterized in that, include: The lexicon building module is used to acquire multiple target words and multiple excluded words, and to build a preset lexicon based on the multiple target words and the multiple excluded words. The target words include one or more target characters, and the excluded words are non-target words formed by concatenating a first string composed of one or more target characters and a second string composed of one or more non-target characters. The structure building module is used to construct a target word tree structure based on the preset lexicon, wherein each branch of the target word tree structure represents a target word or an excluded word, and each node in the branch structure corresponds to a character in the target word or the excluded word; An attribute configuration module is used to configure node attribute information for each node in the target word tree structure to obtain a target word detection model. The node attribute information represents the attribute information of the character corresponding to the current node and the attribute information of the characters corresponding to the child nodes of the current node. The node attribute information includes pronunciation attribute information, shape attribute information, wildcard information, and exclusion attribute information. The target word detection model is configured to: during the detection process, if there is no node in the first-level nodes of the target word detection model that precisely matches a character in the text to be detected, determine whether a matching first-level node exists sequentially based on the pronunciation attribute information, shape attribute information, and wildcard information; in subsequent matching, distinguish between excluded words and target words based on the exclusion attribute information, and obtain the target word detection result after traversal.
13. A target word detection device, characterized in that, include: The acquisition module is used to acquire the text to be detected, which includes multiple characters; The first execution module is used to perform a first specified step for the i-th character in the text to be detected. The first specified step includes: determining whether there is a first-level node in the preset target word detection model that matches the i-th character. The first-level node is a child node of the root node of the target word detection model. The second execution module is used to cyclically execute a second specified step starting from the (i+1)th character when a first-level node matching the i-th character exists in the target word detection model. The second specified step includes: determining whether the j-th character matches the child node of the node matching the (j-1)-th character; if a match is found, determining whether a preset condition is met based on the exclusion attribute information corresponding to the j-th character; if the preset condition is not met, updating j to j+1. The preset condition includes: the j-th character is the end of the target word, and the child node of the j-th character does not contain any non-target characters, or the j-th character is the end of an exclusion word. The end of the target word refers to the target character at the end of the target word, and the end of the exclusion word refers to the non-target character at the end of the exclusion word, i≥1, j≥i+1. The result determination module is used to obtain the detection results of the i-th to j-th characters until the preset conditions are met based on the exclusion attribute information corresponding to the j-th character, and to execute the first specified step for the j+1-th character; The total result acquisition module is used to obtain the target word detection result after completing the detection of all characters in the text to be detected that are within the preset detection range; The target word detection model is obtained using the construction method of the target word detection model as described in any one of claims 1-3.
14. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the method for constructing a target word detection model as described in any one of claims 1-3, or the target word detection method as described in any one of claims 4-10.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for constructing the target word detection model as described in any one of claims 1-3, or the target word detection method as described in any one of claims 4-10.
Citation Information
Patent Citations
Sensitive word filtering method and system
CN105574090A
Sensitive word detection method and device, computer equipment, storage medium and product
CN115391524A