A method, device, equipment and storage medium for quickly searching for sensitive words in text

Abstract

The present invention discloses a method, device, equipment and storage medium for rapid retrieval of sensitive words in a text, which belongs to the field of data security technology. The method comprises the following steps: predefining multiple levels of sensitive words, setting different levels of desensitization rules for sensitive words of different levels, establishing a sensitive word library and expanding it to obtain a sensitive word extension library; normalizing a text to be retrieved, dividing the text to be retrieved into multiple segments using a text block function, and recording the starting position and ending position of each segment of text; multiple task nodes reading a sensitive word list, performing sensitive word retrieval on assigned subtexts, and recording the position of the sensitive words relative to overlapping areas; and calculating the total number of sensitive words in the text based on the number of matches between non-overlapping areas and overlapping areas counted by all task nodes. The present invention adopts multiple nodes to perform sensitive word retrieval operations in parallel, with each node only responsible for sensitive word position marking and quantity statistics of the corresponding subtext, thereby improving retrieval efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and in particular to a method, device, equipment and storage medium for quickly searching for sensitive words in a text. Background Art

[0002] The volume of information disseminated online is increasing, and the need for quality monitoring of online information is rapidly increasing. To foster a civilized and harmonious internet environment, existing online text monitoring typically involves searching and filtering for "sensitive words." These include banned words and uncivilized language as listed by Xinhua News Agency.

[0003] In the prior art, the patent publication number is CN115438140A, and its name is a sensitive word retrieval method and system based on suffix index. The method includes looping through each character in the target text, then building a common prefix array of the target text, and then searching the common prefix array through the multi-level suffix index of the pre-built sensitive word library to obtain the retrieval results of the sensitive words in the target text. However, the shortcomings of this method are: the text is not standardized, which easily leads to the possibility of misjudgment of vocabulary text, thereby reducing efficiency and accuracy. At the same time, the pre-built model also has requirements for the format of the text, and cannot be used for large-scale normalized text modeling processing, which will lead to loss of retrieval accuracy.

[0004] Another example is patent application number CN202210476382.X, titled "A Method for Sensitive Word Analysis Based on a Cloud Computing Platform." Its technical solution is as follows: Specific steps for sensitive word analysis are as follows: 1) Read text; 2) Analyze the text and extract individual words; 3) Determine whether the word is a sensitive word; if so, proceed to step 5); if not, proceed to the next step; 4) Determine whether the meaning of the word is a sensitive word; if so, proceed to the next step; if not, proceed to step 6); 5) Mark sensitive words or words containing sensitive meanings in the text; 6) End sensitive word analysis. This method has shortcomings in that it relies too much on cloud computing, the sensitive word identification method is overly simple, and it also lacks processing of the original text. The construction of sensitive words relies on a sensitive word library, but the specifications of the sensitive word library are not well explained. The cloud platform big data method is used to generalize the retrieval of sensitive words.

[0005] Another example is the patent publication number CN113157904A, and the patent name is a sensitive word filtering method and system based on the DFA algorithm. The sensitive word filtering method includes: aggregating sensitive words to construct a sensitive word library; generating a sensitive word search tree based on the sensitive word library; searching the text content to be retrieved in the sensitive word search tree in the order of the text to obtain all the sensitive words contained in the text content. The shortcomings of this method are: due to the use of the DFA algorithm and the construction of a mapping relationship for sensitive words, when the text is too long or there are cyclic sensitive words, it will have a significant impact on performance. At the same time, if the sensitive word library is too large and the constructed Map mapping is too large, it will also affect efficiency. As a result, the overall efficiency of the implementation of the method is low.

[0006] The sensitive word retrieval methods in the above-mentioned prior art each have their own shortcomings. In particular, if there are too many sensitive words in the sensitive word list, a large number of sensitive words will match texts with small data volumes, resulting in excessively high retrieval overhead and the inability to achieve fast retrieval while ensuring retrieval accuracy. In view of this, the present invention provides a method for quickly retrieving sensitive words in text. Summary of the Invention

[0007] The present invention aims to solve the problem that traditional retrieval methods cannot guarantee retrieval accuracy and complete retrieval quickly when there are too many sensitive words. A method, device, equipment and storage medium for rapid retrieval of sensitive words in a text are proposed. This method can improve the efficiency of sensitive word retrieval when there are too many sensitive words in the sensitive word list.

[0008] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention is as follows:

[0009] A method for quickly searching for sensitive words in a text, characterized by comprising the following steps:

[0010] Predefine multiple levels of sensitive words, set different levels of desensitization rules for different levels of sensitive words, establish a sensitive word library and expand it to obtain a sensitive word extension library;

[0011] Normalize the text to be retrieved, use the text segmentation function to divide the text into multiple sections, and record the starting and ending positions of each section of text;

[0012] Multiple task nodes read the sensitive word list, perform sensitive word search on the assigned subtext, and record the position of sensitive words relative to the overlapping area;

[0013] The total number of sensitive words in the text is calculated based on the number of matches in the non-overlapping and overlapping areas counted by all task nodes.

[0014] Furthermore, the sensitive word library is expanded, including: adding a set of alternative words formed by mixing Chinese and English, mixing pinyin and Chinese characters, and abbreviations of existing sensitive words to the existing sensitive word library.

[0015] Furthermore, the normalization of the text to be retrieved includes: removing irrelevant symbols, logos and possible special characters in the text to delete unnecessary tags.

[0016] Furthermore, when using the text block function to divide the text to be retrieved into multiple segments, the overlapping area of ​​adjacent text sub-blocks must be able to contain the longest sensitive word in the sensitive word library, and the number of blocks cannot exceed the number of parallel task nodes.

[0017] Furthermore, the sensitive word library is expanded to include: combining words with the same prefix in the sensitive words to generate a tree structure, and the next character position of the current character is selected by hashing.

[0018] Furthermore, the plurality of task nodes read the sensitive word list and perform sensitive word retrieval on the assigned subtext, including:

[0019] First, the parallel task node matches the invalid character library with the text to be retrieved and removes the invalid characters;

[0020] Then, based on the prefix information tree structure generated in the sensitive word extension library, the sensitive words in the sensitive word list are scanned and matched with the text;

[0021] Finally, the sensitive word extension library is scanned synchronously with the text, and Chinese and English replacements, pinyin Chinese characters, and abbreviations are also matched and recorded.

[0022] The present invention also provides a device for quickly searching for sensitive words in a text, comprising:

[0023] A sensitive word database, which is used to store predefined multi-level sensitive words and corresponding desensitization rules, as well as a set of alternative words formed by mixing Chinese and English, mixing pinyin and Chinese characters, and abbreviations of existing sensitive words; and combining words with the same prefix in sensitive words to form a tree structure, and selecting the next character position of the current character using a hash method;

[0024] A text preprocessing module is used to normalize the text to be retrieved, divide the text to be retrieved into multiple sections using a text segmentation function, and record the starting position and ending position of each section of text;

[0025] A parallel task processing module, wherein the parallel task processing module is used for multiple task nodes to read the sensitive word list, perform sensitive word search on the assigned subtext, and record the position of the sensitive word relative to the overlapping area;

[0026] The sensitive word statistics module is used to calculate the total number of sensitive words in the text based on the number of matches between non-overlapping areas and overlapping areas counted by all task nodes.

[0027] Furthermore, the parallel task processing module is used for each task node to first match the invalid character library with the text to be retrieved and remove invalid characters; then, based on the prefix information tree structure generated in the sensitive word extension library, the sensitive words in the sensitive word list are scanned and matched with the search text; finally, the sensitive word extension library and the text to be retrieved are scanned synchronously, and Chinese and English replacements, pinyin Chinese characters, and abbreviations are also matched and recorded.

[0028] The present invention also provides a device for quickly retrieving sensitive words in a text, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, the steps in the method for quickly retrieving sensitive words in a text as described above are implemented.

[0029] A computer-readable storage medium stores a computer program, and running the computer program can execute the steps of the above-mentioned method for quickly retrieving sensitive words in a text.

[0030] In summary, the present invention has the following advantages:

[0031] 1) The sensitive word rapid retrieval method of the present invention performs block processing on the sensitive word text, retaining the overlapping area to ensure that the sub-text will not be missed due to the segmentation of sensitive word retrieval;

[0032] 2) The sensitive word rapid retrieval method of the present invention uses multiple nodes to perform sensitive word retrieval operations in parallel. Each node is only responsible for marking the position of sensitive words in the corresponding subtext and counting the number, thereby improving retrieval efficiency.

[0033] 3) The sensitive word rapid retrieval method of the present invention forms a set of alternative words for sensitive words by expanding sensitive words, including abbreviations, abbreviations, and Chinese and English replacements, thereby avoiding the situation where sensitive words are filtered due to existing operation forms. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of the rapid retrieval of sensitive words in the present invention. DETAILED DESCRIPTION

[0035] In order to more clearly illustrate the present invention, the present invention is further described below in conjunction with preferred embodiments and drawings. Those skilled in the art should understand that the following specific description is illustrative rather than restrictive and should not be used to limit the scope of protection of the present invention.

[0036] The present invention provides a method for quickly searching for sensitive words in a text, comprising the following steps:

[0037] Step 1: Sensitive word entry: The role uses the data desensitization system to build a sensitive word library.

[0038] First, the administrator of the data desensitization platform predefines multiple levels of sensitive words and presets different desensitization rules for sensitive words at different levels; then, sensitive words are added to the corresponding levels, and operations such as modification and deletion of sensitive words are performed according to actual needs, thereby establishing a sensitive word library.

[0039] Step 2: Expand the sensitive word library.

[0040] In response to possible abbreviations and Chinese-English conversions of sensitive words, the sensitive word library is expanded, and a set of alternative words formed by mixing Chinese and English, mixing pinyin and Chinese characters, and abbreviations of existing sensitive words are added to the existing sensitive word library.

[0041] Specifically, the operations for expanding the sensitive word library include:

[0042] 1) Initialize an invalid string library, such as special characters such as space, @, #, etc., to remove invalid characters in the text to be detected.

[0043] 2) Unify the English capitalization and abbreviations of the sensitive word library, and convert between pinyin and Chinese characters to form an extended character library.

[0044] 3) Obtain a list of sensitive words from the sensitive word library, combine the words with the same prefix in the sensitive words to generate a tree structure, and select the next character position of the current character using a hash method.

[0045] Trie(S all )→(Hash,Key,Value)

[0046] Among them, S all is a set of sensitive words, Hash is the storage method, Key is the key of the hash map, Value is the specific value, and may also be the next layer {Key, Value} in the prefix structure.

[0047] Step 3: Normalize the text to be retrieved, remove irrelevant symbols, logos, and possible special characters in the text, and delete unnecessary tags.

[0048] 1) Read the sensitive word list in the sensitive word library and use the text segmentation function to divide the text to be retrieved into multiple segments:

[0049] K=[P+L max (N-1)]mod N;

[0050] Among them, P is the text size, L max is the longest sensitive word length in the sensitive word library, N is the number of parallel task nodes, K is the size of the text block, and the value of the overlapping area is greater than or equal to L max

[0051] It should be noted that the overlapping area of ​​adjacent text sub-blocks must be able to contain the longest sensitive word in the sensitive word library, and the number of blocks cannot exceed the number of parallel task nodes.

[0052] 2) Record the starting and ending positions of each part of the text:

[0053] Si=(i-1)(KL max );

[0054] Ei=i(KL max );

[0055] Among them S i Indicates the left boundary of the i-th subtext, E i Indicates the right margin of the i-th subtext.

[0056] Step 4: Multiple task nodes read the sensitive word list, perform sensitive word search on the assigned sub-text, and record the position of the sensitive words relative to the overlapping area; mark them differently according to the level of the sensitive words, and distinguish whether the sensitive words are located in the overlapping area.

[0057] The specific steps include:

[0058] First, the parallel task node Node i Match the invalid character library generated in step 2 with the text to remove invalid characters;

[0059] Then, based on the prefix information tree structure generated in step 2, the sensitive words in the sensitive word list are scanned and matched with the text:

[0060] MatchSWord(Trie,P i )→{S word ,left,right}

[0061] Among them, Trie is a tree prefix structure, P i For the i-th subtext, {S word ,left,right} represent the detected sensitive words and their specific locations.

[0062] Similarly, the sensitive word extension library generated in step 2 is scanned synchronously with the text, and Chinese and English replacements, pinyin Chinese characters, abbreviations, etc. are also matched and recorded.

[0063] Step 5: Merger stage.

[0064] The total number of sensitive words in the text is calculated based on the number of matches in the non-overlapping and overlapping areas counted by all task nodes. The sensitive word location statistics for each task node are aggregated. If a sensitive word is completely within the overlapping area, it is counted only once; all other sensitive words are counted normally.

[0065] The following is a specific example to illustrate this method.

[0066] Role information: Data desensitization platform administrator (User B), text sensitive word search role (User C), parallel task node i .

[0067] Follow these steps to implement this method:

[0068] Data desensitization platform initialization;

[0069] User B divides the sensitive words into different levels and enters them to form a sensitive word set S all ;

[0070] Get the list of sensitive words and combine the sensitive words to generate a prefix tree structure Trie (S all ), each node actually saves {Key, Value} in a hashing manner, where the Key is the current node and the Value is the value corresponding to the node, which may be the last letter of a word or the next {Key, Value} key-value pair;

[0071] User C reads the sensitive word list from the sensitive word library and uses the text chunking function to obtain the text chunk size K and the corresponding position;

[0072] Parallel task node i According to the prefix information tree structure Trie (S all ), match sensitive words with text and record sensitive word location information {S word ,left,right};

[0073] User C calculates the total number of sensitive words in the text based on the number of matches in the non-overlapping and overlapping areas counted by all task nodes.

[0074] Example 2

[0075] This embodiment provides a device for quickly searching for sensitive words in a text, including: a sensitive word database, a text preprocessing module, a parallel task processing module, and a sensitive word statistics module.

[0076] The sensitive word database is used to store predefined multi-level sensitive words and corresponding desensitization rules, as well as a set of alternative words formed by mixing Chinese and English, mixing pinyin and Chinese characters, and abbreviations of existing sensitive words; further, the sensitive word database combines words with the same prefix in sensitive words to generate a tree structure, and the next character position of the current character is selected using a hash method.

[0077] The text preprocessing module is used to perform normalization processing on the text to be retrieved, divide the text to be retrieved into multiple sections using a text block function, and record the starting position and the ending position of each part of the text.

[0078] Specifically, the text preprocessing module normalizes the text to be retrieved by removing irrelevant symbols, logos, and possible special characters in the text to delete unnecessary tags.

[0079] The parallel task processing module is used for multiple task nodes to read the sensitive word list, perform sensitive word search on the assigned sub-text, and record the position of the sensitive words relative to the overlapping area.

[0080] Specifically, when the parallel task processing module is running, each task node first matches the invalid character library with the text to be retrieved to remove invalid characters; then, based on the prefix information tree structure generated in the sensitive word database, the sensitive words in the sensitive word list are scanned and matched with the text to be retrieved; finally, the sensitive word database and the text to be retrieved are scanned synchronously, and Chinese and English replacements, pinyin Chinese characters, and abbreviations are also matched and recorded.

[0081] The sensitive word statistics module is used to calculate the total number of sensitive words in the text based on the number of matches between the non-overlapping areas and the overlapping areas counted by all task nodes.

[0082] Example 3

[0083] The present invention also provides a device for quickly retrieving sensitive words in a text, which can be used to implement a method for quickly retrieving sensitive words in a text as described in the aforementioned embodiment. The device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When executing the computer program, the processor implements the steps in a method for quickly retrieving sensitive words in a text as described in the aforementioned embodiment.

[0084] Preferably, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units can be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the device.

[0085] The processor can be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor. The processor is the control center of the device, and uses various interfaces and lines to connect the various parts of the device.

[0086] The memory mainly includes a program storage area and a data storage area. The program storage area can store the operating system, at least one application required for a function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory such as a plug-in hard disk, a smart memory card, a secure digital card, and a flash memory card, or the memory can also be other volatile solid-state memory devices.

[0087] Example 4

[0088] An embodiment of the present invention further discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a method for quickly retrieving sensitive words in a text as disclosed in the above embodiment 1 is implemented.

[0089] Computer storage media may be tangible media that can contain or store programs for use by or in connection with an instruction execution system, apparatus, or device.

[0090] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification or equivalent change made to the above embodiment based on the technical essence of the present invention shall fall within the scope of protection of the present invention.

Claims

1. A method for quickly searching for sensitive words in a text, characterized in that: The steps include: Predefine multiple levels of sensitive words, set different levels of desensitization rules for different levels of sensitive words, establish a sensitive word library and expand it to obtain a sensitive word extension library; Normalize the text to be retrieved and use the text segmentation function to divide it into multiple segments. Record the starting and ending positions of each segment. When using the text segmentation function to divide the text to be retrieved into multiple segments, the overlapping areas of adjacent text sub-blocks must be able to contain the longest sensitive word in the sensitive word library. The number of segments cannot exceed the number of parallel task nodes. Multiple task nodes read the sensitive word list, perform sensitive word search on the assigned subtext, and record the position of sensitive words relative to the overlapping area; Calculate the total number of sensitive words in the text based on the number of matches in the non-overlapping and overlapping areas counted by all task nodes; The multiple task nodes read the sensitive word list and perform sensitive word retrieval on the assigned subtext, including: First, the parallel task node matches the invalid character library with the text to be retrieved and removes the invalid characters; Then, based on the prefix information tree structure generated in the sensitive word extension library, the sensitive words in the sensitive word list are scanned and matched with the text; Finally, the sensitive word extension library is scanned synchronously with the text, and Chinese and English replacements, pinyin Chinese characters, and abbreviations are also matched and recorded.

2. The method for quickly searching for sensitive words in a text according to claim 1, characterized in that: Expand the sensitive word library, including adding a set of alternative words formed by mixing Chinese and English, mixing pinyin and Chinese characters, and abbreviations of existing sensitive words to the existing sensitive word library.

3. The method for quickly searching for sensitive words in a text according to claim 1, characterized in that: The normalization of the text to be retrieved includes: clearing irrelevant symbols, logos and special characters in the text to delete unnecessary tags.

4. The method for quickly searching for sensitive words in a text according to claim 2, wherein: Expanding the sensitive word library also includes: combining words with the same prefix in the sensitive words to generate a tree structure, and selecting the next character position of the current character using a hash method.

5. A device for quickly searching for sensitive words in a text, characterized in that: include: A sensitive word database, which is used to store predefined multi-level sensitive words and corresponding desensitization rules, as well as a set of alternative words formed by mixing Chinese and English, mixing pinyin and Chinese characters, and abbreviations of existing sensitive words; and combining words with the same prefix in sensitive words to form a tree structure, and selecting the next character position of the current character using a hash method; A text preprocessing module is used to normalize the text to be retrieved, divide the text to be retrieved into multiple segments using a text segmentation function, and record the starting and ending positions of each segment. After the text is segmented, the overlapping area of ​​adjacent text sub-blocks contains the longest sensitive word in the sensitive word library, and the number of segments does not exceed the number of parallel task nodes. A parallel task processing module, wherein the parallel task processing module is used for multiple task nodes to read the sensitive word list, perform sensitive word search on the assigned subtext, and record the position of the sensitive word relative to the overlapping area; Multiple task nodes read the sensitive word list and perform sensitive word searches on the assigned subtexts. The process involves: each task node first matches the invalid character library with the text to be searched, removing invalid characters; then, based on the prefix information tree structure generated in the sensitive word extension library, it scans and matches the sensitive words in the sensitive word list with the search text; finally, the sensitive word extension library and the text to be searched are scanned synchronously, matching and recording Chinese-English substitutions, pinyin Chinese characters, and abbreviations. The sensitive word statistics module is used to calculate the total number of sensitive words in the text based on the number of matches between non-overlapping areas and overlapping areas counted by all task nodes.

6. A device for quickly searching for sensitive words in a text, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing a computer program, wherein the computer program can be executed to execute the steps of the method according to any one of claims 1 to 4.

Images