Sensitive Word Detection Method, Apparatus, Device, and Computer-Readable Storage Medium

Abstract

This application relates to the field of artificial intelligence, and provides a sensitive word detection method, device, equipment and computer-readable storage medium to detect sensitive words of various complex sensitive types and / or custom sensitive types in text. The method includes: expanding a basic sensitive word library based on a sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules; detecting a target detection text based on a non-sensitive word library, a stop word library and the sensitive word expansion library; if all the keywords detected from the target detection text are non-sensitive words, determining that the target detection text is a non-sensitive text; if no non-sensitive words are detected from the target detection text, preprocessing the target detection text and continuing to detect the preprocessed target detection text. The technical solution of this application has significant advantages such as high detection rate, low false alarm rate and high recall rate.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to a method, device, equipment and computer-readable storage medium for detecting sensitive words. Background Art

[0002] The application field of sensitive word detection is becoming wider and wider. Especially today with the popularization of the Internet, sensitive word detection is an effective technical means to combat various illegal behaviors such as online porn, cult-related and prohibited-related. At present, in related technologies, the AC automaton, that is, the Aho-Corasick automaton, is a commonly used sensitive word detection method. However, when illegal actors carry out illegal acts, they often distort the words in a text, resulting in the AC automaton being unable to detect them. In other words, the existing technologies cannot detect the sensitive content of texts with complex sensitive types and custom-sensitive types. Summary of the Invention

[0003] This application provides a method, device, equipment and computer-readable storage medium for detecting sensitive words to detect sensitive words or combinations of sensitive words of various complex sensitive types and / or custom-sensitive types in a text.

[0004] On the one hand, this application provides a method for detecting sensitive words, including:

[0005] Expanding a basic sensitive word library based on a sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1;

[0006] Detecting a target detection text based on a non-sensitive word library, a stop word library and the sensitive word expansion library;

[0007] If all the keywords detected from the target detection text are non-sensitive words, it is determined that the target detection text is a non-sensitive text;

[0008] If no non-sensitive words are detected from the target detection text, preprocess the target detection text and continue to detect the preprocessed target detection text.

[0009] On the other hand, this application provides a device for detecting sensitive words, including:

[0010] An expansion module, configured to expand a basic sensitive word library based on a sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1;

[0011] The first detection module is used to detect the target detection text based on the non-sensitive word library, the stop word library, and the sensitive word expansion library.

[0012] The determination module is used to determine that the target detection text is non-sensitive if all the keywords detected from the target detection text are non-sensitive words.

[0013] The second detection module is used to preprocess the target detection text and continue to detect the preprocessed target detection text if no non-sensitive words are detected from the target detection text.

[0014] In a third aspect, the present application provides a device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the technical solution of the sensitive word detection method as described above are implemented.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the technical solution of the sensitive word detection method as described above are implemented.

[0016] As can be seen from the technical solutions provided by the present application above, since the libraries relied on for detecting the target detection text include not only the non-sensitive word library and the stop word library, but also the sensitive word expansion library obtained by expanding the basic sensitive word library based on the sensitive word expansion algorithm, the technical solutions provided by the present application can detect variants such as shape similarity, pronunciation similarity, traditional Chinese characters, pinyin, synonyms, antonyms, or reverse order, that is, can detect sensitive words with multiple sensitive types and complex combinations, and have significant advantages such as high detection rate, low false alarm rate, and high recall rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 is a flowchart of the sensitive word detection method provided by the embodiment of the present application;

[0019] Figure 2 is a schematic structural diagram of the sensitive word detection device provided by the embodiment of the present application;

[0020] Figure 3 is a schematic structural diagram of the device provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0022] In this specification, adjectives such as first and second can only be used to distinguish one element or action from another element or action, and do not necessarily require or imply any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0023] In this specification, for ease of description, the dimensions of the various parts shown in the drawings are not drawn in actual proportional relationship.

[0024] Refer to Figure 1 , which is the sensitive word detection method flow provided by the embodiments of the present application, mainly includes steps S101 to S104, and is described in detail as follows:

[0025] Step S101: Expand the basic sensitive word library based on the sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1.

[0026] In the embodiments of the present application, the basic sensitive word library is a word library composed of keywords of specified sensitive types combined by specific rules. Since illegal actors involved in porn, cults, and prohibited items, etc. will use various variants of sensitive words, bypassing the basic sensitive word library, therefore, the basic sensitive word library can be expanded based on the sensitive word expansion algorithm to obtain a sensitive word expansion library, and then the target detection text can be detected based on the sensitive word expansion library. As an embodiment of the present application, expanding the basic sensitive word library based on the sensitive word expansion algorithm to obtain a sensitive word expansion library can be: setting the target number of target sensitive words and their variants, where the target sensitive word is the sensitive word in the basic sensitive word library that needs to be expanded; according to the word-variant sequence dictionary and the target number of target sensitive words and their variants, expanding the target sensitive words based on the principle of permutation and combination to obtain a sensitive word expansion library. In the above embodiment, the word-variant sequence dictionary is created according to the word library inside the sensitive word expansion algorithm, and the variants of the target sensitive word include words that are similar in shape, pronunciation to the target sensitive word, as well as the traditional Chinese, pinyin, synonyms, antonyms, reverse order, etc. of the target sensitive word. The variants of the above target sensitive words can be permuted and combined based on the principle of permutation and combination, so as to expand the target sensitive word and obtain a sensitive word expansion library.

[0027] It should be noted that the multi-sensitive word pairs included in the sensitive word expansion library are composed of entity sensitive words and emotional sensitive words, or composed of entity sensitive words, emotional sensitive words, and negative words. The combination rules for single sensitive words and / or multi-sensitive word pairs that make up I types of sensitive types include separating different single sensitive words or multi-sensitive word pairs with line breaks, connecting entity sensitive words, emotional sensitive words, or negative words within multi-sensitive word pairs with semicolons, connecting entity sensitive words and emotional sensitive words with plus signs, connecting emotional sensitive words and negative words with minus signs, and so on. In addition, the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I types of sensitive types, and I is an integer greater than 1, indicating that there are multiple sensitive types.

[0028] Step S102: Detect the target detection text based on the non-sensitive word library, the stop word library, and the sensitive word expansion library.

[0029] In the embodiment of the present application, the target detection text is the text to be detected, that is, the text for which it is necessary to determine whether it contains sensitive words or whether it is sensitive text. Specifically, the implementation of step S102 can be: read the non-sensitive word library, stop word library, and sensitive word expansion library, and input the target detection text; for different sensitive types and non-sensitive types to be detected in the sensitive word expansion library, establish a detection object respectively; for each detection object, add the corresponding sensitive word list to the AC automaton, and create their respective trie trees; then, based on the trie trees, detect the target detection text, that is, the AC automaton matches the sensitive words included in the trie trees with the target detection text to determine whether all the keywords detected from the target detection text are non-sensitive words.

[0030] Step S103: If all the keywords detected from the target detection text are non-sensitive words, determine that the target detection text is non-sensitive text.

[0031] If, after step S102, all the keywords detected from the target detection text are non-sensitive words, determine that the target detection text is non-sensitive text. It should be noted here that if all the keywords detected from the target detection text are non-sensitive words, it is of course possible to directly determine that the target detection text is non-sensitive text. However, conversely, if no non-sensitive words have been detected in the target detection text, it cannot be determined whether the target detection text is non-sensitive text. In this case, further detection is required, that is, the process enters the subsequent step S104.

[0032] Step S104: If no non-sensitive words have been detected from the target detection text, preprocess the target detection text and continue to detect the preprocessed target detection text.

[0033] As described above, if all the keywords detected from the target detection text are non-sensitive words, it can be directly determined that the target detection text is non-sensitive. However, various factors can lead to the inability to detect non-sensitive words from the target detection text. Therefore, if no non-sensitive words are detected from the target detection text, it does not mean that the target detection text is non-sensitive, and further detection of the target detection text is required. In the embodiments of the present application, if no non-sensitive words are detected from the target detection text, the target detection text is preprocessed, and the preprocessed target detection text is continuously detected. Specifically, it can be: using delimiters to split the target detection text into sentences, obtaining J sentences; taking I sensitive types as target sensitive types, and detecting sensitive words for each of the J sentences one by one until a combination of sensitive words is output or an empty result is output, where J is an integer greater than or equal to 1. Until a combination of sensitive words is output or an empty result is output, it means that a combination of sensitive words is output, or, because no sensitive words are actually detected from the target detection text, an empty result is output. In the above embodiments, using delimiters to split the target detection text into sentences to obtain J sentences actually includes two cases. Specifically, it is to determine whether the total number of characters in the target detection text is greater than a preset value (for example, 100 characters). If so, the target detection text is split into sentences with Chinese or English semicolons, exclamation marks, question marks, colons, Chinese full stops, and line breaks as delimiters; if the total number of characters in the target detection text is not greater than the preset value (for example, 100 characters), the target detection text is split into sentences with line breaks as delimiters.

[0034] As for taking I sensitive types as target sensitive types and detecting sensitive words for each of the J sentences one by one until a combination of sensitive words is output or an empty result is output, as an embodiment of the present application, it can be implemented through steps S11 to S18, and the detailed description is as follows:

[0035] Step S11: Taking the i-th sensitive type among the I sensitive types as the target sensitive type, and using the first sensitive word detection algorithm to detect sensitive words in the j-th sentence among the J sentences, where i = 1, 2,..., I and j = 1, 2,..., J.

[0036] Here, i is any value among 1, 2,..., I, which means that the implementation of the technical solution of the present application can start from any one of the I sensitive types as the target sensitive type; without loss of generality, it can start from i = 1 and use the first sensitive word detection algorithm to detect sensitive words in the j-th sentence among the J sentences; similarly, for the value of j, it means that the implementation of the technical solution of the present application can start from any one of the J sentences for sensitive word detection; without loss of generality, it can start from j = 1, that is, the first sentence, to detect sensitive words in the target detection text.

[0037] Step S12: If a sensitive word is detected from the j-th clause, stopword processing is performed on the j-th clause to obtain the j-th clause after stopword processing.

[0038] Performing stopword processing on a clause means filtering out words (such as modal particles, adverbs, prepositions, conjunctions in Chinese, and "i", "is", "a", "the", etc. in English) that have little value for detecting sensitive words in the clause using a computer program or manual means, so that the words remaining in the final clause are all words with practical meanings. Performing stopword processing on a clause can not only significantly reduce the workload of subsequent processes, but also improve the accuracy of sensitive word detection due to fewer interfering variables.

[0039] Step S13: If a sensitive word is detected from the j-th clause after stopword processing, the second sensitive word detection algorithm is used to perform sensitive word detection on the j-th clause after stopword processing again.

[0040] Step S14: If a sensitive word is detected when performing sensitive word detection on the j-th clause after stopword processing again using the second sensitive word detection algorithm, the sensitive word detection result is retained.

[0041] Since step S14 only performs sensitive word detection on the j-th clause after stopword processing again and has not completed the sensitive word detection of all J clauses, it is necessary to retain the sensitive word detection result for further processing after all J clauses have completed sensitive word detection.

[0042] Step S15: After incrementing j by 1, jump to step S11.

[0043] After incrementing j by 1 and jumping to step S11, it means that with the i-th sensitive type among the I sensitive types as the target sensitive type, the first sensitive word detection algorithm is used to restart the sensitive word detection of the (j + 1)-th clause among the J clauses.

[0044] Step S16: Loop through the above steps S11 to S15 until after all J clauses are detected, increment i by 1 and then jump to step S11.

[0045] Looping through the above steps S11 to S15 means that with the target sensitive type unchanged (with the i-th sensitive type as the target sensitive type), according to the technical solutions of steps S11 to S15, each of the J clauses is subjected to sensitive word detection one by one until the value of j increases to J.

[0046] Step S17: Loop through the above steps S11 to S16 until the sensitive word detection of the J clauses with the i-th sensitive type as the target sensitive type is completed.

[0047] Repeating the above steps S11 to S17 means that, with the detection object remaining unchanged (J clauses), according to the technical solutions of steps S11 to S17, the target sensitive type is changed (incrementing i by 1) to perform sensitive word detection on the J clauses until the value of i increases to I.

[0048] Step S18: Combine the sensitive word detection results retained after each loop above and output the combined sensitive word result.

[0049] As described above, whenever step S14 is passed, the sensitive word detection result is retained; after the J clauses are detected with I sensitive types as the target sensitive types, the sensitive word detection results retained after each loop above are combined and the combined sensitive word result is output. It should be noted that if no sensitive word is detected after each loop, an empty result is output.

[0050] As an embodiment of the present application, the above sensitive word detection of the j-th clause among the J clauses using the first sensitive word detection algorithm can be implemented through steps S21 to S27, which are described in detail as follows:

[0051] Step S21: Use the AC automaton to perform sensitive word detection on the j-th clause to obtain a sensitive word list.

[0052] In the embodiment of the present application, the sensitive word list lists the results of sensitive word detection on the j-th clause using the AC automaton, that is, which detected sensitive words are included.

[0053] Step S22: If the ratio of the number of characters in the sensitive word list to the number of characters in the j-th clause is greater than a preset value, combine the sensitive word list into a sensitive word detection result dictionary; otherwise, output an empty dictionary.

[0054] For example, if the ratio of the number of characters in the sensitive word list to the number of characters in the j-th clause is greater than a preset value, such as 50 (or other preset values), the sensitive word list can be combined into a sensitive word detection result dictionary; otherwise, an empty dictionary is output.

[0055] Step S23: If the character length of each sensitive word in the sensitive word detection result dictionary is not greater than 2, is an English string, or is a numeric string, execute step S24; otherwise, execute step S27.

[0056] Step S24: Segment the j-th clause to obtain a segmentation list.

[0057] Step S25: If the segmentation list contains the sensitive words in the sensitive word detection result dictionary, execute step S26; otherwise, execute step S27.

[0058] Step S26: Return the sensitive word detection result dictionary.

[0059] Here, return the sensitive word detection result dictionary, that is, output the sensitive word detection result dictionary.

[0060] Step S27: Determine that the sensitive word detection result is a non-sensitive word.

[0061] As an embodiment of the present application, the above-mentioned use of the second sensitive word detection algorithm to perform sensitive word detection on the j-th processed clause with stop words again can be: creating a two-dimensional list based on the sensitive word expansion library and the non-sensitive word library; based on the Q sensitive words in the sensitive word list and the combination of P sensitive words in the two-dimensional list, performing loop detection on the j-th processed clause with stop words until a sensitive word is detected, where the sensitive word list is the list obtained by using the first sensitive word detection algorithm to perform sensitive word detection on the j-th clause among the J clauses in the above-mentioned embodiment. It should be noted that in the above-mentioned embodiment, each row of the two-dimensional list corresponds to a sensitive word combination respectively, and each column of the two-dimensional list corresponds to a set of sensitive types. Correspondingly, creating a two-dimensional list based on the sensitive word expansion library and the non-sensitive word library can be: placing the single sensitive words in the sensitive word expansion library into the first set of sensitive types of the two-dimensional list, and setting the second set of sensitive types and the third set of sensitive types of the two-dimensional list to be empty; or placing the entity words in the multi-sensitive word pairs in the sensitive word expansion library into the first set of sensitive types of the two-dimensional list, and placing the emotional words in the multi-sensitive word pairs into the second set of sensitive types of the two-dimensional list; and if there are negative words in the multi-sensitive word pairs, placing these negative words into the third set of sensitive types of the two-dimensional list. In the above-mentioned embodiment, the first set of sensitive types, the second set of sensitive types, or the third set of sensitive types respectively represents a combination type of sensitive words, that is, a sensitive word combination. It can be understood that the two-dimensional list may include more than the above three sets of sensitive types, and may also include more sets of sensitive types. Taking the two-dimensional list including three sets of sensitive types as an example, an example of the two-dimensional list created based on the sensitive word expansion library and the non-sensitive word library is as follows:

[0062]

[0063] A more specific example (only for reference, not real data) is as follows:

[0064] Two-dimensional list = [[{New York, San Francisco, Boston,...}, {High housing prices, dirty and messy,...}, {Not, Isn't,...}], [{Amazon, NetFlix,...}, {Many fakes, Slow delivery,...}, {}], [{falungong}, {}, {}],...].

[0065] Regarding the combination of Q sensitive words in the sensitive word list and P sensitive words in the two-dimensional list in the above embodiments, the j-th deactivated word processing clause is cyclically detected until a sensitive word is detected. Specifically, it can be implemented through steps S301 to S314. The following takes an example where a sensitive word combination contains three sensitive type sets (the first sensitive type set, the second sensitive type set, and the third sensitive type set) and is described in detail as follows:

[0066] Step S301: Start from the p-th sensitive word combination in the created two-dimensional list.

[0067] Step S302: Start from the q-th sensitive word in the sensitive word list obtained in step S21 of the above embodiment.

[0068] Step S303: If the second sensitive type set in the p-th sensitive word combination is empty, execute step S304; otherwise, jump to step S305.

[0069] Step S304: Determine whether the q-th sensitive word is included in the first sensitive type set of the p-th sensitive word combination. If so, return the result "true"; otherwise, jump to step S314.

[0070] Step S305: If the third sensitive type set of the p-th sensitive word combination is empty, execute step S306; otherwise, jump to step S309.

[0071] Step S306: Determine whether the q-th sensitive word is included in the first sensitive type set of the p-th sensitive word combination. If so, increment count_11 by 1, that is, record count_11 = count_11 + 1.

[0072] Step S307: Determine whether the q-th sensitive word is included in the second sensitive type set of the p-th sensitive word combination. If so, increment count_12 by 1, that is, record count_12 = count_12 + 1.

[0073] Step S308: Determine whether count_11>0 and count_12>0 hold. If so, return the result "true"; otherwise, jump to step S315.

[0074] Step S309: Determine whether the q-th sensitive word is included in the first sensitive type set of the p-th sensitive word combination. If so, increment count_21 by 1, that is, record count_21 = count_21 + 1.

[0075] Step S310: Determine whether the q-th sensitive word is included in the second sensitive type set of the p-th sensitive word combination. If so, increment count_22 by 1, that is, record count_22 = count_22 + 1.

[0076] Step S311: Determine whether the q-th sensitive word is included in the third sensitive type set of the p-th sensitive word combination. If so, increment count_23 by 1, i.e., set count_23 = count_23 + 1.

[0077] Step S312: Determine whether count_21 > 1, count_22 > 1, and count_23 > 1 are all true. If so, return the result "true"; otherwise, jump to step S315.

[0078] Step S313: Determine whether the q-th sensitive word is the last sensitive word in the sensitive word list obtained in step S21. If so, jump to step S315; otherwise, increment q by 1 (i.e., q = q + 1) and jump to step 302.

[0079] Step S314: Determine whether the p-th sensitive word combination is the last sensitive word combination in the created two-dimensional list. If so, jump to step S315; otherwise, increment p by 1 (i.e., p = p + 1) and jump to step 301.

[0080] Step S315: Return the result "false".

[0081] From the above-described sensitive word detection method of the Figure 1 example, it can be seen that since the libraries relied on for detecting the target detection text include not only the non-sensitive word library and the stop word library, but also the sensitive word expansion library obtained by expanding the basic sensitive word library based on the sensitive word expansion algorithm, the technical solution provided by this application can detect variants such as similar in form, similar in pronunciation, traditional Chinese, pinyin, synonyms, antonyms, or reverse order, that is, it can detect sensitive words with multiple sensitive types and complex combinations, and has significant advantages such as high detection rate, low false alarm rate, and high recall rate.

[0082] Please refer to the Figure 2 , which is a sensitive word detection device provided by an embodiment of this application, and may include an expansion module 201, a first detection module 202, a determination module 203, and a second detection module 204, which are described in detail as follows:

[0083] The expansion module 201 is configured to expand the basic sensitive word library based on the sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1;

[0084] The first detection module 202 is configured to detect the target detection text based on the non-sensitive word library, the stop word library, and the sensitive word expansion library;

[0085] A determination module 203, configured to determine that the target detection text is non-sensitive text if all the keywords detected from the target detection text are non-sensitive words;

[0086] A second detection module 204, configured to preprocess the target detection text if no non-sensitive words are detected from the target detection text, and continue to detect the preprocessed target detection text.

[0087] From the above-mentioned Figure 2 Example sensitive word detection device, it can be seen that since the libraries relied on for detecting the target detection text include not only the non-sensitive word library and the stop word library, but also the sensitive word expansion library obtained by expanding the basic sensitive word library based on the sensitive word expansion algorithm, the technical solution provided by this application can detect variants such as similar in shape, similar in pronunciation, traditional Chinese, pinyin, synonyms, antonyms or reverse order, that is, it can detect sensitive words with multiple sensitive types and complex combinations, and has significant advantages such as high detection rate, low false alarm rate and high recall rate.

[0088] Figure 3 It is a schematic structural diagram of a device provided by an embodiment of the present application. As Figure 3 shown, the device 3 of this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for the sensitive word detection method. When the processor 30 executes the computer program 32, it implements the steps in the above-mentioned sensitive word detection method embodiment, such as Figure 1 the steps S101 to S104 shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-mentioned device embodiments, such as Figure 2 the functions of the expansion module 201, the first detection module 202, the determination module 203, and the second detection module 2045 shown.

[0089] Exemplarily, the computer program 32 of the sensitive word detection method mainly includes: expanding the basic sensitive word library based on the sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1; detecting the target detection text based on the non-sensitive word library, the stop word library, and the sensitive word expansion library; if the keywords detected from the target detection text are all non-sensitive words, determining that the target detection text is a non-sensitive text; if no non-sensitive words are detected from the target detection text, preprocessing the target detection text and continuing to detect the preprocessed target detection text. The computer program 32 can be divided into one or more modules / units, and one or more modules / units are stored in the memory 31 and executed by the processor 30 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the device 3. For example, the computer program 32 can be divided into the functions of an expansion module 201, a first detection module 202, a determination module 203, and a second detection module 204 (modules in the virtual device), and the specific functions of each module are as follows: The expansion module 201 is used to expand the basic sensitive word library based on the sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1; the first detection module 202 is used to detect the target detection text based on the non-sensitive word library, the stop word library, and the sensitive word expansion library; the determination module 203 is used to determine that the target detection text is a non-sensitive text if the keywords detected from the target detection text are all non-sensitive words; the second detection module 204 is used to preprocess the target detection text and continue to detect the preprocessed target detection text if no non-sensitive words are detected from the target detection text.

[0090] The device 3 may include but is not limited to the processor 30 and the memory 31. Those skilled in the art can understand that Figure 3 merely examples of the device 3, which do not constitute a limitation on the device 3, may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the computing device may also include input / output devices, network access devices, buses, etc.

[0091] The so-called processor 30 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0092] The memory 31 may be an internal storage unit of the device 3, such as the hard disk or memory of the device 3. The memory 31 may also be an external storage device of the device 3, such as a plug-in hard disk equipped on the device 3, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 31 may also include both the internal storage unit of the device 3 and the external storage device. The memory 31 is used to store computer programs and other programs and data required by the device. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In practical applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working process of the units and modules in the above device can refer to the corresponding process in the foregoing method embodiment and will not be elaborated herein.

[0094] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0095] Those of ordinary skill in the art will realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0096] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in an electrical, mechanical or other form.

[0097] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0099] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-transitory computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by instructing relevant hardware through a computer program. The computer program of the sensitive word detection method can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above various method embodiments, that is, based on a sensitive word expansion algorithm, expand the basic sensitive word library to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1; based on the non-sensitive word library, the stop word library, and the sensitive word expansion library, detect the target detection text; if all the keywords detected from the target detection text are non-sensitive words, determine that the target detection text is a non-sensitive text; if no non-sensitive words are detected from the target detection text, preprocess the target detection text and continue to detect the preprocessed target detection text. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The non-transitory computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the non-transitory computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the non-transitory computer-readable medium does not include electrical carrier signals and telecommunication signals. The above embodiments are only used to illustrate the technical solutions of this application, not to limit it; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included in the protection scope of this application. The above-described specific implementation manners further elaborate on the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above is only the specific implementation manner of this application and is not used to limit the protection scope of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of this application should all be included in the protection scope of this invention.

Claims

1. A sensitive word detection method, characterized in that, The method includes: Expanding a basic sensitive word library based on a sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1; Detecting a target detection text based on a non-sensitive word library, a stop word library, and the sensitive word expansion library; If all the keywords detected from the target detection text are non-sensitive words, determining that the target detection text is a non-sensitive text; If no non-sensitive words are detected from the target detection text, preprocessing the target detection text and continuing to detect the preprocessed target detection text; The preprocessing the target detection text and continuing to detect the preprocessed target detection text includes: Using a delimiter to split the target detection text into J clauses, where J is an integer greater than or equal to 1; Taking the I sensitive types as target sensitive types, and performing sensitive word detection on each of the J clauses one by one until a sensitive word combination result is output or an empty result is output.

2. The sensitive word detection method according to claim 1, wherein The expanding a basic sensitive word library based on a sensitive word expansion algorithm to obtain a sensitive word expansion library includes: Setting a target quantity of a target sensitive word and its variants, where the target sensitive word is a sensitive word in the basic sensitive word library that needs to be expanded; Based on a word-variant sequence dictionary and the target quantity of the target sensitive word and its variants, expanding the target sensitive word based on the principle of permutation and combination to obtain the sensitive word expansion library.

3. The sensitive word detection method according to claim 1, wherein The taking the I sensitive types as target sensitive types and performing sensitive word detection on each of the J clauses one by one until a sensitive word combination result is output or an empty result is output includes: Step S11: Taking the i-th sensitive type of the I sensitive types as the target sensitive type, and using a first sensitive word detection algorithm to perform sensitive word detection on the j-th clause among the J clauses, where i = 1, 2,..., I and j = 1, 2,..., J; Step S12: If a sensitive word is detected from the j-th clause, performing stop word processing on the j-th clause to obtain a j-th clause after stop word processing; Step S13: If a sensitive word is detected from the j-th clause after stop word processing, using a second sensitive word detection algorithm to perform sensitive word detection on the j-th clause after stop word processing again; Step S14: If a sensitive word is detected when using the second sensitive word detection algorithm to perform sensitive word detection on the j-th clause after stop word processing again, retaining the sensitive word detection result; Step S15: Incrementing j by 1 and then jumping to step S11; Step S16: Repeating the above steps S11 to S15 until after all the J clauses are detected, incrementing i by 1 and then jumping to step S11; Step S17: Repeating the above steps S11 to S16 until the detection of the J clauses is completed with the I sensitive types as the target sensitive types; Step S18: Combining the retained sensitive word detection results after each of the above loops and outputting the sensitive word combination result.

4. The sensitive word detection method according to claim 3, wherein Performing sensitive word detection on the j-th clause among the J clauses using the first sensitive word detection algorithm includes: Step S21: Performing sensitive word detection on the j-th clause using an AC automaton to obtain a sensitive word list; Step S22: If the ratio of the number of characters in the sensitive word list to the number of characters in the j-th clause is greater than a preset value, combining the sensitive word list into a sensitive word detection result dictionary; otherwise, outputting an empty dictionary; Step S23: If the character length of each sensitive word in the sensitive word detection result dictionary is no more than 2, is an English string, or is a numeric string, then execute Step S24; otherwise, execute Step S27; Step S24: Segmenting the j-th clause to obtain a word segmentation list; Step S25: If the word segmentation list contains the sensitive words in the sensitive word detection result dictionary, then execute Step S26; otherwise, execute Step S27; Step S26: Returning the sensitive word detection result dictionary; Step S27: Determining that the sensitive word detection result is a non-sensitive word.

5. The sensitive word detection method according to claim 4, characterized in that, Performing sensitive word detection on the j-th clause after stop word processing again using the second sensitive word detection algorithm includes: Creating a two-dimensional list based on the sensitive word expansion library and the non-sensitive word library; Performing loop detection on the j-th clause after stop word processing based on the combination of Q sensitive words in the sensitive word list and P sensitive words in the two-dimensional list until a sensitive word is detected, where the sensitive word list is the list obtained by performing sensitive word detection on the j-th clause among the J clauses using the first sensitive word detection algorithm.

6. The sensitive word detection method according to claim 5, wherein Each row of the two-dimensional list corresponds to a sensitive word combination respectively, and each column of the two-dimensional list corresponds to a set of sensitive types. Creating the two-dimensional list based on the sensitive word expansion library includes: Placing the single sensitive words in the sensitive word expansion library into the first set of sensitive types in the two-dimensional list, and setting the second and third sets of sensitive types in the two-dimensional list to be empty; or Placing the entity words in the multi-sensitive word pairs in the sensitive word expansion library into the first set of sensitive types in the two-dimensional list, and placing the sentiment words in the multi-sensitive word pairs into the second set of sensitive types in the two-dimensional list; If there is a negative word in the multi-sensitive word pair, placing the negative word into the third set of sensitive types in the two-dimensional list.

7. A sensitive word detection device, characterized in that, The device includes: An expansion module for expanding a basic sensitive word library based on a sensitive word expansion algorithm to obtain a sensitive word expansion library, where the sensitive word expansion library includes single sensitive words and / or multi-sensitive word pairs of I sensitive types formed according to different combination rules, and I is an integer greater than 1; A first detection module for detecting a target detection text based on a non-sensitive word library, a stop word library, and the sensitive word expansion library; A determination module for determining that the target detection text is a non-sensitive text if all the keywords detected from the target detection text are non-sensitive words; A second detection module, configured to preprocess the target detection text and continue to detect the preprocessed target detection text if no non-sensitive word has been detected from the target detection text; the preprocessing of the target detection text and the continued detection of the preprocessed target detection text include: Using a delimiter to split the target detection text into J clauses, where J is an integer greater than or equal to 1; Taking the I sensitive types as target sensitive types, and performing sensitive word detection on each of the J clauses one by one until a sensitive word combination result is output or an empty result is output.

8. A device, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images