A sensitive word retrieval method and system based on suffix index

Abstract

The present invention discloses a sensitive word retrieval method and system based on suffix index. The method comprises looping through each character in a target text, then constructing a common prefix array of the target text, and then searching the common prefix array through a multi-level suffix index of a pre-constructed sensitive word library to obtain retrieval results of sensitive words in the target text, thereby improving retrieval speed and reducing space overhead.

Description

Technical Field

[0001] The present invention relates to the field of sensitive word retrieval, and in particular to a sensitive word retrieval method and system based on suffix indexing. Background Art

[0002] The volume of information disseminated online is currently increasing, and the demand for monitoring the quality of online information is correspondingly rapidly increasing. To foster a civilized and harmonious internet environment, existing online text monitoring typically relies on searching and filtering for "sensitive words." These include banned words and uncivilized language as announced by Xinhua News Agency. However, current methods for searching for sensitive words suffer from the following issues: Machine learning and artificial intelligence-based methods require significant time to train models; methods based on text content require additional processing such as word segmentation and feature extraction; and methods based on tree or graph structures incur significant space overhead. Summary of the Invention

[0003] The present invention provides a sensitive word retrieval method and system based on suffix index, which improves retrieval speed and reduces space overhead.

[0004] In order to achieve the above object, the present invention provides a sensitive word retrieval method based on suffix index, comprising:

[0005] By looping through each text character of the target text, the traversed text character is added to the back of each common prefix in the common prefix array of the target text;

[0006] According to a preset suffix index retrieval algorithm, through a preset retrieval method, each common prefix of the target text is retrieved based on a multi-level suffix index in a pre-constructed sensitive word library; the multi-level suffix index is a plurality of suffix indexes corresponding to a plurality of sensitive word sub-libraries obtained by dividing the sensitive word library through a preset partitioning algorithm; the suffix index stores the index number of the suffix of each sensitive word arranged in ascending order of characters;

[0007] If the corresponding common prefix of the target text cannot be retrieved according to the multi-level suffix index, deleting the common prefix from the common prefix array of the target text;

[0008] If a corresponding common prefix is retrieved according to the multi-level suffix index, the sensitive words of the target text are obtained according to the index number stored in the suffix index.

[0009] The present invention provides a sensitive word retrieval method based on suffix index. First, each text character of an input target text is traversed in a loop, and the text character is added to each common prefix in a common prefix array of the target text, so as to avoid data loss in the detection process and incomplete retrieval. At the same time, new common prefixes are continuously added in the form of a loop traversal, so as to avoid waste of space resources. Then, the common prefixes are searched through a multi-level suffix index in a pre-constructed sensitive word library. Searching according to the constructed multi-level suffix index reduces the retrieval time. At the same time, the multi-level suffix index is a plurality of suffix indexes of a plurality of sensitive word sub-libraries divided, so as to reduce space overhead. Then, the common prefixes are searched based on the constructed multi-level suffix index. If the corresponding common prefix cannot be detected, it is deleted to avoid repeated checking. If the corresponding common prefix is detected, the retrieval result of the sensitive word is obtained according to the index number stored in the suffix index, so as to improve the accuracy of the retrieval.

[0010] As a preferred example, the step of adding the traversed text characters to the back of each common prefix in the common prefix array of the target text specifically includes:

[0011] By adding a common prefix with a word length of 0 to the common prefix array, according to each text character of the target text traversed, adding the same text character to the corresponding common prefix in the common prefix array;

[0012] The common prefix is a character string whose length is less than or equal to the length of the sensitive word and is stored in the common prefix array which is initially empty.

[0013] The present invention avoids data loss by looping through each character of the input target text. At the same time, each traversed character is added to the corresponding common prefix according to the loop, thereby avoiding waste of space resources and reducing space overhead. At the same time, a simple retrieval basis is provided for subsequent retrieval, thereby reducing retrieval time.

[0014] As a preferred example, the process of constructing the multi-level suffix index of the sensitive word library specifically includes:

[0015] By traversing the sensitive word library and adding a terminator after each sensitive word, a statistical result of the number of sensitive words of each word length in the sensitive word library is obtained;

[0016] According to the statistical results, the sensitive word library is divided by a preset division algorithm to obtain a plurality of sensitive word sub-libraries;

[0017] By using a preset suffix index construction algorithm, according to the plurality of sensitive word sub-libraries, a plurality of suffix indexes corresponding to each of them are obtained;

[0018] The metadata of the plurality of suffix indexes are saved in a preset data saving method to obtain a multi-level suffix index of the sensitive word library, wherein the metadata includes the minimum word length, the maximum word length and the offset of the character bucket of the sensitive word.

[0019] The present invention divides the sensitive word library to obtain several sensitive word sub-libraries, which can support subsequent retrieval according to the division rules, reducing the retrieval time, and then constructs a suffix index for the sensitive word sub-library according to a preset suffix index algorithm. The suffix index corresponds to the various metadata included in the sensitive word, so that the retrieval criteria become richer and the retrieval accuracy is improved.

[0020] As a preferred example, in the above-mentioned method of searching for each common prefix of the target text according to the multi-level suffix index in the pre-built sensitive word library based on the preset suffix index retrieval algorithm and the preset retrieval method, the method specifically includes:

[0021] The preset suffix index retrieval algorithms include sequential method, binary method and LF-Mapping;

[0022] The preset retrieval methods include step-by-step serial retrieval, parallel retrieval at all levels, and distributed retrieval;

[0023] Sensitive word retrieval is performed on each common prefix of the target text through each suffix index in the multi-level suffix index in the pre-built sensitive word library according to a preset suffix index retrieval algorithm and retrieval method.

[0024] The present invention provides several suffix index retrieval algorithms and retrieval methods, providing multiple options, and then searching each common prefix through each suffix index in the multi-level suffix index in the pre-built sensitive word library to improve the retrieval speed.

[0025] As a preferred example, the step of retrieving the corresponding common prefix according to the multi-level suffix index specifically includes:

[0026] In the process of storing the index numbers of the suffixes of each sensitive word arranged in ascending order of characters through the suffix index, suffixes with the same first character are clustered in the suffix index to obtain continuous character buckets.

[0027] When forming the suffix index, the present invention groups suffixes with the same characters together to form character buckets. During the search process, the existence of the character buckets can reduce the search time and improve the search speed.

[0028] As a preferred example, obtaining the sensitive words of the target text specifically includes:

[0029] If a common prefix corresponding to the target text is retrieved according to the multi-level suffix index, and the next character is detected as a terminator in the character bucket included in the suffix index corresponding to the common prefix, the sensitive word is saved in the search result, and the common prefix corresponding to the sensitive word is deleted from the common prefix array;

[0030] If the corresponding common prefix in the target text is retrieved according to the multi-level suffix index, but the next character is not detected as the end character in the character bucket contained in the suffix index corresponding to the common prefix, the sensitive word is not saved in the search result, and the common prefix corresponding to the sensitive word is not deleted from the common prefix array.

[0031] During the retrieval process, the present invention needs to first detect the corresponding common prefix, and then determine whether there is an end symbol in the corresponding character bucket, thereby improving the retrieval standard and thus improving the accuracy of the retrieval. At the same time, if the corresponding common prefix is retrieved and the corresponding end symbol is detected at the same time, it will be deleted, which avoids repeated checks on the one hand and avoids occupying space resources on the other hand, thereby reducing space overhead.

[0032] As a preferred example, the step of storing the sensitive words specifically includes:

[0033] Saving the sensitive words to the search results through a preset saving method;

[0034] The storage methods include arrays, linked lists and queues.

[0035] The present invention provides a variety of ways to save data so that the retrieved sensitive words can be saved in the search results.

[0036] On the other hand, the present invention provides a sensitive word retrieval system based on suffix index, comprising a loop module, a retrieval module, a deletion module and a storage module;

[0037] The loop module is used for looping through each text character of the target text and adding the traversed text character to the back of each common prefix in the common prefix array of the target text;

[0038] The retrieval module is configured to retrieve each common prefix of the target text according to a preset suffix index retrieval algorithm and a preset retrieval method based on a multi-level suffix index in a pre-constructed sensitive word library; the multi-level suffix index is a plurality of suffix indexes corresponding to a plurality of sensitive word sub-libraries obtained by dividing the sensitive word library using a preset partitioning algorithm; the suffix index stores the index number of each sensitive word suffix arranged in ascending order of characters;

[0039] The deletion module is configured to delete the common prefix from the common prefix array of the target text if the corresponding common prefix of the target text cannot be retrieved according to the multi-level suffix index;

[0040] The storage module is configured to obtain the sensitive words of the target text according to the index number stored in the suffix index if a corresponding common prefix is retrieved according to the multi-level suffix index.

[0041] The present invention provides a sensitive word retrieval system based on a suffix index. First, a loop module is used to loop through each text character of an input target text, and the text character is added to each common prefix in a common prefix array of the target text, thereby avoiding data loss during the detection process, resulting in incomplete retrieval. At the same time, common prefixes are continuously added in the form of loop traversal, thereby avoiding waste of space resources. Then, a retrieval module is used to search for the common prefix according to a multi-level suffix index in a pre-constructed sensitive word library. Searching according to the constructed multi-level suffix index reduces retrieval time. At the same time, the multi-level suffix index is a plurality of suffix indexes of a plurality of sensitive word sub-libraries divided, thereby reducing space overhead. Then, the common prefix is searched based on the constructed multi-level suffix index. If a deletion module determines that a corresponding common prefix cannot be detected, it is deleted to avoid repeated checks. If a storage module determines that a corresponding common prefix is detected, the retrieval result of the sensitive word is obtained according to the index number stored in the multi-level suffix index, thereby improving the accuracy of the retrieval.

[0042] As a preferred example, the retrieval module includes a construction unit and a retrieval unit;

[0043] The construction unit includes traversing the sensitive word library and adding a terminator after each sensitive word to obtain statistical results of the number of sensitive words of each word length in the sensitive word library; dividing the sensitive word library using a preset division algorithm based on the statistical results to obtain a plurality of sensitive word sub-libraries; using a preset suffix index construction algorithm, obtaining a plurality of suffix indexes corresponding to each of the sensitive word sub-libraries; saving metadata of the plurality of suffix indexes using a preset data saving method to obtain a multi-level suffix index of the sensitive word library, wherein the metadata includes the minimum word length, the maximum word length, and the offset of the character bucket of the sensitive word;

[0044] The retrieval unit is used to perform sensitive word retrieval for each common prefix of the target text through each suffix index in the multi-level suffix index in the pre-built sensitive word library according to a preset suffix index retrieval algorithm and retrieval method; the preset suffix index retrieval algorithm includes a sequential method, a binary method and LF-Mapping; the preset retrieval method includes a level-by-level serial retrieval, a parallel retrieval at all levels and a distributed retrieval.

[0045] The present invention divides the sensitive word library through a construction unit to obtain several sensitive word sub-libraries, which can support subsequent retrieval according to the division rules, reducing the retrieval time, and then constructs a suffix index for the sensitive word sub-library according to a preset suffix index algorithm. The suffix index corresponds to various metadata included in the sensitive word, so that the retrieval standard becomes richer and the accuracy of the retrieval is improved. Each suffix index in the multi-level suffix index in the pre-constructed sensitive word library by the retrieval unit is used to search for each common prefix, thereby improving the retrieval speed.

[0046] As a preferred example, the saving module includes a saving unit and a neglecting unit;

[0047] The saving unit is configured to save the sensitive word in the search result and delete the common prefix corresponding to the sensitive word from the common prefix array if a common prefix corresponding to the target text is retrieved according to the multi-level suffix index and the next character is detected as a terminator in the character bucket included in the suffix index corresponding to the common prefix;

[0048] The ignoring unit is used for not saving the sensitive word in the search result and not deleting the common prefix corresponding to the sensitive word from the common prefix array if the corresponding common prefix in the target text is retrieved according to the multi-level suffix index, but the next character is not detected as a terminator in the character bucket contained in one of the suffix indexes corresponding to the common prefix.

[0049] The present invention saves the retrieved sensitive words and saves them in the search results through the saving unit, and deletes the common prefixes corresponding to the sensitive words from the common prefix array, thereby avoiding the waste of space resources. At the same time, the common prefixes for which the corresponding end symbols are not retrieved are neither deleted nor saved through the ignoring unit, and the search can be repeated in the loop process, thereby improving the accuracy of the search. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 : A flowchart of a sensitive word retrieval method based on suffix indexing provided by an embodiment of the present invention;

[0051] Figure 2: A structural diagram of a sensitive word retrieval system based on suffix indexing provided by an embodiment of the present invention;

[0052] Figure 3 : A structural diagram of a sensitive word retrieval system based on suffix indexing provided by another embodiment of the present invention;

[0053] Figure 4 : A schematic diagram of a multi-level suffix index construction process provided by another embodiment of the present invention;

[0054] Figure 5 : A schematic diagram of a sensitive word search process provided by another embodiment of the present invention;

[0055] Figure 6 : A schematic diagram of a sensitive word search result provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] Example 1

[0058] Please refer to Figure 1 , which is a flow chart of a sensitive word search method based on suffix indexing provided by an embodiment of the present invention, mainly includes steps 101 to 104, specifically including:

[0059] Step 101: traverse each text character of the target text in a loop, and add the traversed text character to the back of each common prefix in the common prefix array of the target text.

[0060] In this embodiment, this step is specifically as follows: by adding a new common prefix with a length of 0 to the common prefix array, according to each text character of the target text traversed, the same text character is added to the corresponding common prefix in the common prefix array; the common prefix is a character string with a length less than or equal to the length of the sensitive word, and is stored in the common prefix array which is initially empty.

[0061] Step 102: According to a preset suffix index retrieval algorithm and a preset retrieval method, each common prefix of the target text is retrieved according to the multi-level suffix index in the pre-built sensitive word library.

[0062] In this embodiment, this step is specifically as follows: by traversing the sensitive word library and adding a terminator after each sensitive word, a statistical result of the number of sensitive words of each word length in the sensitive word library is obtained; based on the statistical result, the sensitive word library is divided by a preset division algorithm to obtain several sensitive word sub-libraries; by using a preset suffix index construction algorithm, according to the several sensitive word sub-libraries, several suffix indexes corresponding to each are obtained; metadata of the several suffix indexes is saved by a preset data saving method to obtain a multi-level suffix index of the sensitive word library, where the metadata includes the minimum word length, maximum word length and character bucket offset of the sensitive word.

[0063] In this embodiment, the preset suffix index retrieval algorithm includes sequential method, binary method and LF-Mapping; the preset retrieval method includes step-by-step serial retrieval, parallel retrieval at all levels and distributed retrieval; each suffix index in the multi-level suffix index in the pre-built sensitive word library is used to perform sensitive word retrieval for each common prefix of the target text according to the preset suffix index retrieval algorithm and retrieval method.

[0064] Step 103: If the corresponding common prefix of the target text cannot be retrieved according to the multi-level suffix index, the common prefix is deleted from the common prefix array of the target text.

[0065] Step 104: If a corresponding common prefix is retrieved according to the multi-level suffix index, the sensitive words of the target text are obtained according to the index number stored in the suffix index.

[0066] In this embodiment, the step is specifically as follows: if the corresponding common prefix of the target text is retrieved according to the multi-level suffix index, and the next character is detected as a terminator in the character bucket contained in the one suffix index corresponding to the common prefix, the sensitive word is saved in the search result, and the common prefix corresponding to the sensitive word is deleted from the common prefix array; if the corresponding common prefix in the target text is retrieved according to the multi-level suffix index, but the next character is not detected as a terminator in the character bucket contained in the one suffix index corresponding to the common prefix, the sensitive word is not saved in the search result, and the common prefix corresponding to the sensitive word is not deleted from the common prefix array.

[0067] In this embodiment, the suffix index stores the index numbers of the suffixes of each sensitive word, which are arranged in ascending order according to character sequence. Suffixes with the same first character are clustered in the suffix index to obtain continuous character buckets. The sensitive words are stored in the search results using a preset storage method, which includes an array, a linked list, and a queue.

[0068] Please refer to Figure 2 , which is a sensitive word retrieval system based on suffix index provided by this embodiment, which mainly includes a loop module 201, a retrieval module 202, a deletion module 203 and a storage module 204.

[0069] The loop module 201 is used to loop through each text character of the target text and add the traversed text character to the end of each common prefix in the common prefix array of the target text.

[0070] The retrieval module 202 is configured to retrieve each common prefix of the target text according to a preset suffix index retrieval algorithm and a preset retrieval method based on a multi-level suffix index in a pre-built sensitive word library; the multi-level suffix index is a plurality of suffix indexes corresponding to a plurality of sensitive word sub-libraries obtained by dividing the sensitive word library using a preset partitioning algorithm; the suffix index stores the index number of each sensitive word suffix arranged in ascending order of character sequence;

[0071] The deleting module 203 is configured to delete the common prefix from the common prefix array of the target text if the corresponding common prefix of the target text cannot be retrieved according to the multi-level suffix index.

[0072] The storage module 204 is configured to obtain the sensitive words of the target text according to the index number stored in the suffix index if a corresponding common prefix is retrieved according to the multi-level suffix index.

[0073] In this embodiment, the retrieval module 202 includes a construction unit and a retrieval unit.

[0074] The construction unit includes traversing the sensitive word library and adding a terminator after each sensitive word to obtain statistical results of the number of sensitive words of each word length in the sensitive word library; dividing the sensitive word library by a preset division algorithm according to the statistical results to obtain a plurality of sensitive word sub-libraries; obtaining a plurality of suffix indexes corresponding to the plurality of sensitive word sub-libraries by using a preset suffix index construction algorithm; and saving metadata of the plurality of suffix indexes by a preset data saving method to obtain a multi-level suffix index of the sensitive word library, wherein the metadata includes the minimum word length, the maximum word length and the offset of the character bucket of the sensitive word.

[0075] The retrieval unit is used to perform sensitive word retrieval for each common prefix of the target text through each suffix index in the multi-level suffix index in the pre-built sensitive word library according to a preset suffix index retrieval algorithm and retrieval method; the preset suffix index retrieval algorithm includes a sequential method, a binary method and LF-Mapping; the preset retrieval method includes a level-by-level serial retrieval, a parallel retrieval at all levels and a distributed retrieval.

[0076] In this embodiment, the saving module 204 includes a saving unit and a ignoring unit.

[0077] The saving unit is used to save the sensitive word in the search result and delete the common prefix corresponding to the sensitive word from the common prefix array if the corresponding common prefix in the target text is retrieved according to the multi-level suffix index and the next character is detected as a terminator in the character bucket contained in one of the suffix indexes corresponding to the common prefix.

[0078] The ignoring unit is used for not saving the sensitive word in the search result and not deleting the common prefix corresponding to the sensitive word from the common prefix array if the corresponding common prefix in the target text is retrieved according to the multi-level suffix index, but the next character is not detected as a terminator in the character bucket contained in one of the suffix indexes corresponding to the common prefix.

[0079] Example 2

[0080] Please refer to Figure 3 , which is a structural diagram of a sensitive word retrieval system based on suffix indexing provided by another embodiment of the present invention.

[0081] The present embodiment provides a sensitive word search system based on suffix index, which includes a suffix index construction module and a sensitive word search module. The system input includes a sensitive word library and a target text; the system output includes a suffix index and a search result.

[0082] Please refer to Figure 4 , which is a schematic diagram of a multi-level suffix index construction process according to another embodiment of the present invention, mainly includes steps 401 to 404, specifically including:

[0083] Step 401: traverse the sensitive word library, add a terminator '$' after each sensitive word, and count the number of sensitive words of various word lengths.

[0084] In this embodiment, the step is specifically as follows: traverse the sensitive word library, add a terminator '$' after each sensitive word, and count the number of sensitive words with a length of 2, 3, and 4 in the sensitive word library to be 1 each.

[0085] Step 402: Based on the statistical results of the number of sensitive words of various lengths, the sensitive word library is divided into two sensitive word sub-libraries of comparable size according to the length of the sensitive words.

[0086] In this embodiment, the step is specifically as follows: based on the statistical results of the number of sensitive words, the sensitive word library is divided into two sensitive word sub-libraries of equivalent size according to the length of the sensitive words, and the sensitive word sub-library 1 is divided into: "ab$aab$"; sensitive word sub-library 2 is: "bcca$".

[0087] Step 403: construct a suffix index for each sensitive word sub-library using an inductive replication (sorting) suffix index construction algorithm in a serial manner.

[0088] In this embodiment, the step is specifically: using the inductive replication (sorting) suffix index construction algorithm in a serial manner to construct a level 1 suffix index: {6, 2, 3, 4, 0, 5, 1}; a level 2 suffix index: {4, 3, 0, 2, 1}.

[0089] Step 404: Save the suffix index metadata of each sensitive word sub-library in an array format to obtain a multi-level suffix index of the sensitive word library.

[0090] In this embodiment, this step specifically involves saving the suffix index metadata for each sensitive word sub-library in the form of an array. The metadata format is: {minimum word length, maximum word length, {character bucket offset}}. The first-level suffix index metadata is: {2, 3, {$: 0, a: 2, b: 5}}; the second-level suffix index metadata is: {4, 4, {$: 0, a: 1, b: 2, c: 3}}.

[0091] Please refer to Figure 5 , is a sensitive word search flow chart provided by another embodiment of the present invention, which mainly includes steps 501 to 507, specifically including:

[0092] Step 501: Starting from the first character in the target text, traverse each text character.

[0093] In this embodiment, this step specifically includes: looping through each text character of the target text, starting from the first character 'c' of the target text, and traversing each text character one by one.

[0094] Step 502: Add a new common prefix with a word length of 0 to the common prefix array, and add the currently traversed text character to the end of each common prefix in the common prefix array.

[0095] In this embodiment, this step is specifically as follows: adding the traversed text characters to the end of each common prefix in the common prefix array of the target text to obtain a current common prefix array: {c}.

[0096] Step 503: Search each common prefix in the common prefix array in a multi-level suffix index of the sensitive vocabulary using the LF-Mapping suffix index search algorithm in a step-by-step serial manner.

[0097] In this embodiment, the specific steps are as follows: according to a preset suffix index retrieval algorithm, through a preset retrieval method, there is no matching character bucket in the first-level suffix index metadata for the current common prefix 'c' in the common prefix array, and therefore there is no need to use the LF-Mapping suffix index retrieval algorithm to retrieve the first-level suffix index; there is a matching character bucket in the second-level suffix index metadata, and therefore the LF-Mapping suffix index retrieval algorithm is used to retrieve the second-level suffix index starting from the offset '3' of the character bucket.

[0098] Step 504: If the current common prefix cannot be retrieved in the multi-level suffix index, it is deleted from the common prefix array.

[0099] In this embodiment, the step is specifically as follows: the current common prefix 'c' can be retrieved in the level 2 suffix index, and therefore does not need to be deleted from the common prefix array.

[0100] Step 505: If the current common prefix is retrieved in the multi-level suffix index, and there is an item whose next character is the end character in the matching character bucket, the sensitive word is recorded in the search result and deleted from the common prefix array.

[0101] In this embodiment, the specific step is: the current common prefix 'c' can be retrieved in the second-level suffix index, but there is no item in the matching character bucket whose next character is the end character, so there is no need to record the finding of the sensitive word in the search results, and there is no need to delete it from the common prefix array.

[0102] Step 506: Check whether the retrieval of all common prefixes is completed, otherwise jump to step 503 to retrieve the next common prefix.

[0103] Step 507: Check whether the traversal of the target text is completed, otherwise jump to step 502 to traverse the next text character.

[0104] In this embodiment, the specific step is: the traversal of the target text "caabc" is not completed, so jump to step 502 to traverse the next text character 'a' to obtain the sensitive word retrieval result of this embodiment, such as Figure 6 , Figure 6 A schematic diagram of sensitive word search results provided by an embodiment of the present invention.

[0105] In this embodiment, a sensitive word retrieval system based on a suffix index is provided. Retrieval is performed according to the sensitive word retrieval system. A multi-level suffix index construction process is then provided. In constructing the multi-level suffix index, this embodiment completes the construction of the multi-level suffix index based on the sensitive word information in the sensitive word library and records the metadata of each level of the suffix index for invocation by the sensitive word retrieval module. Compared with tree- or graph-based methods, the space overhead is relatively low. Then, in a sensitive word retrieval method provided in this embodiment, this embodiment utilizes the multi-level suffix index and its metadata to quickly and accurately retrieve sensitive words from the target text in the sensitive word library. Compared with methods based on machine learning and artificial intelligence, this method does not require a large amount of time to train the model; compared with methods based on text content, it does not require additional processing such as word segmentation and feature extraction.

[0106] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A sensitive word retrieval method based on suffix index, characterized in that: include: By looping through each text character of the target text, the traversed text character is added to the back of each common prefix in the common prefix array of the target text; wherein, by adding a common prefix with a word length of 0 to the common prefix array, according to each text character of the traversed target text, the same text character is added to the corresponding common prefix in the common prefix array; According to a preset suffix index retrieval algorithm, a preset retrieval method is used to retrieve each common prefix of the target text according to a multi-level suffix index in a pre-constructed sensitive word library; the multi-level suffix index is a plurality of suffix indexes corresponding to a plurality of sensitive word sub-libraries obtained by dividing the sensitive word library by a preset partitioning algorithm; the suffix index stores the index number of the suffix of each sensitive word arranged in ascending order according to character sequence; wherein, by traversing the sensitive word library and adding a terminator after each sensitive word, a statistical result of the number of sensitive words of each word length in the sensitive word library is obtained; based on the statistical result, the sensitive word library is divided by a preset partitioning algorithm to obtain a plurality of sensitive word sub-libraries; by using a preset suffix index construction algorithm, a plurality of suffix indexes corresponding to each of the sensitive word sub-libraries are obtained; metadata of the plurality of suffix indexes is stored by a preset data storage method to obtain a multi-level suffix index of the sensitive word library, wherein the metadata includes the minimum word length, the maximum word length and the offset of the character bucket of the sensitive word; If the corresponding common prefix of the target text cannot be retrieved according to the multi-level suffix index, deleting the common prefix from the common prefix array of the target text; If a corresponding common prefix is retrieved according to the multi-level suffix index, the sensitive words of the target text are obtained according to the index number stored in the suffix index.

2. The sensitive word search method based on suffix index according to claim 1, characterized in that: The step of adding the traversed text characters to the back of each common prefix in the common prefix array of the target text specifically includes: The common prefix is a character string whose length is less than or equal to the length of the sensitive word and is stored in the common prefix array which is initially empty.

3. The sensitive word search method based on suffix index according to claim 1, characterized in that: The method of searching for each common prefix of the target text according to the multi-level suffix index in the pre-built sensitive word library based on the preset suffix index retrieval algorithm and the preset retrieval method specifically includes: The preset suffix index retrieval algorithms include sequential method, binary method and LF-Mapping; The preset retrieval methods include step-by-step serial retrieval, parallel retrieval at all levels, and distributed retrieval; Sensitive word retrieval is performed on each common prefix of the target text through each suffix index in the multi-level suffix index in the pre-built sensitive word library according to a preset suffix index retrieval algorithm and retrieval method.

4. The sensitive word search method based on suffix index according to claim 1, characterized in that: Retrieving the corresponding common prefix according to the multi-level suffix index specifically includes: In the process of storing the index numbers of the suffixes of each sensitive word arranged in ascending order of characters through the suffix index, suffixes with the same first character are clustered in the suffix index to obtain continuous character buckets.

5. The sensitive word search method based on suffix index according to claim 4, characterized in that: The obtaining of the sensitive words in the target text specifically includes: If a common prefix corresponding to the target text is retrieved according to the multi-level suffix index, and the next character is detected as a terminator in a character bucket included in a suffix index corresponding to the common prefix, the sensitive word is saved in the search result, and the common prefix corresponding to the sensitive word is deleted from the common prefix array; If the corresponding common prefix in the target text is retrieved according to the multi-level suffix index, but the next character is not detected as the end character in the character bucket contained in the suffix index corresponding to the common prefix, the sensitive word is not saved in the search result, and the common prefix corresponding to the sensitive word is not deleted from the common prefix array.

6. The sensitive word search method based on suffix index according to claim 5, characterized in that: The storing of the sensitive words specifically includes: Saving the sensitive words to the search results through a preset saving method; The storage methods include arrays, linked lists and queues.

7. A sensitive word retrieval system based on suffix index, characterized by: It includes loop module, retrieval module, deletion module and saving module; The loop module is used to loop through each text character of the target text and add the traversed text character to the back of each common prefix in the common prefix array of the target text; wherein, a common prefix with a word length of 0 is added to the common prefix array, and according to each text character of the traversed target text, the same text character is added to the corresponding common prefix in the common prefix array; The retrieval module is used to retrieve each common prefix of the target text according to a preset suffix index retrieval algorithm and a preset retrieval method according to a multi-level suffix index in a pre-built sensitive word library; the multi-level suffix index is a plurality of suffix indexes corresponding to a plurality of sensitive word sub-libraries obtained by dividing the sensitive word library by a preset division algorithm; the suffix index stores the index number of the suffix of each sensitive word arranged in ascending order according to character sequence; wherein, the retrieval module includes a construction unit; the construction unit is used to obtain a statistical result of the number of sensitive words of each word length in the sensitive word library by traversing the sensitive word library and adding an end symbol after each sensitive word. According to the statistical results, the sensitive word library is divided into several sensitive word sub-libraries by a preset division algorithm; according to the several sensitive word sub-libraries, several suffix indexes corresponding to each of them are obtained by using a preset suffix index construction algorithm; metadata of the several suffix indexes are saved by a preset data saving method to obtain a multi-level suffix index of the sensitive word library, wherein the metadata includes the minimum word length, the maximum word length and the offset of the character bucket of each sensitive word; wherein, in the process of saving the index number of the suffix of each sensitive word arranged in ascending order of characters by the suffix index, suffixes with the same first character are clustered in the suffix index to obtain continuous character buckets; The deletion module is configured to delete the common prefix from the common prefix array of the target text if the corresponding common prefix of the target text cannot be retrieved according to the multi-level suffix index; The storage module is configured to obtain the sensitive words of the target text according to the index number stored in the suffix index if a corresponding common prefix is retrieved according to the multi-level suffix index.

8. The sensitive word search system based on suffix index according to claim 7, characterized in that: The retrieval module further includes a retrieval unit; The retrieval unit is used to perform sensitive word retrieval for each common prefix of the target text through each suffix index in the multi-level suffix index in the pre-built sensitive word library according to a preset suffix index retrieval algorithm and retrieval method; the preset suffix index retrieval algorithm includes a sequential method, a binary method and LF-Mapping; the preset retrieval method includes a level-by-level serial retrieval, a parallel retrieval at all levels and a distributed retrieval.

9. The sensitive word search system based on suffix index according to claim 7, characterized in that: The preservation module includes a preservation unit and a neglect unit; The saving unit is configured to save the sensitive word in the search result and delete the common prefix corresponding to the sensitive word from the common prefix array if a common prefix corresponding to the target text is retrieved according to the multi-level suffix index and the next character is detected as a terminator in a character bucket included in a suffix index corresponding to the common prefix; The ignoring unit is used for not saving the sensitive word in the search result and not deleting the common prefix corresponding to the sensitive word from the common prefix array if the corresponding common prefix in the target text is retrieved according to the multi-level suffix index, but the next character is not detected as a terminator in the character bucket contained in one of the suffix indexes corresponding to the common prefix.

Images