Text processing method and device and related equipment

Abstract

The invention provides a text processing method and device and related equipment, and relates to the technical field of data processing. The method comprises the steps of obtaining a to-be-processed text; candidate sensitive words are determined in the to-be-processed text on the basis of the starting characters and the ending characters of all the sensitive words in a sensitive word bank, and the sensitive word bank comprises a sensitive word hash table and the starting characters and the ending characters of all the sensitive words; calculating hash values of the candidate sensitive words, and matching the hash values of the candidate sensitive words with the sensitive word hash table; when the hash values of the candidate sensitive words are successfully matched in the sensitive word hash table, obtaining the sensitive words in the to-be-processed text; and replacing the sensitive words in the to-be-processed text to generate a text without the sensitive words. The sensitive word filtering efficiency and accuracy can be improved.

Description

Background Art

[0002] With the development of Internet technology, in order to maintain a good network environment, reduce the spread of bad information, and protect users from abnormal content, it is necessary to filter and replace content in specific scenarios. Sensitive word filtering technology can effectively identify and process inappropriate speech, inappropriate information, etc. to maintain the health and safety of the network environment.

[0003] In the related technologies, sensitive word filtering technology is mainly based on methods such as keyword matching and regular expressions, but these methods have certain limitations in processing speed and accuracy, especially when faced with large amounts of data, it is difficult to strike a balance between efficiency and accuracy.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present invention provides a text processing method, apparatus and related equipment, which can improve the efficiency and accuracy of sensitive word filtering.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a text processing method is provided, the method comprising: obtaining a text to be processed; determining candidate sensitive words in the text to be processed based on the start characters and end characters of each sensitive word in a sensitive word library, wherein the sensitive word library comprises a sensitive word hash table, and the start characters and end characters of each sensitive word; calculating a hash value of the candidate sensitive word, and matching the hash value of the candidate sensitive word with the sensitive word hash table; when the hash value of the candidate sensitive word successfully matches in the sensitive word hash table, obtaining the sensitive word in the text to be processed; replacing the sensitive words in the text to be processed to generate a text without sensitive words.

[0008] In some embodiments, the sensitive word library is generated in the following manner: a hash sensitive word list is placed in a JAVA project, and sensitive words in the hash sensitive word list are annotated to generate a sensitive word library.

[0009] In some embodiments, placing a hash sensitive word table into a JAVA project includes: creating a sensitive word hash table in the JAVA project; wherein the key of the sensitive word hash table is used to store the hash value of the sensitive word, and the value of the sensitive word hash table is used to store the sensitive word; obtaining at least one sensitive word by loading a sensitive word dictionary; calculating the hash value of each sensitive word, and storing each sensitive word and the hash value corresponding to each sensitive word in the sensitive word hash table.

[0010] In some embodiments, sensitive words in a hash sensitive word table are annotated to generate a sensitive word library, including: creating a first character marker array and a second character marker array; wherein the first character marker array is used to mark the start character of each sensitive word, and the second character marker array is used to mark the end character of each sensitive word; updating the first character marker array and the second character marker array of each sensitive word to mark the start character and the end character of each sensitive word; based on the sensitive word hash table, the updated first character marker array of the start character of each sensitive word and the second character marker array of the end character, generating a sensitive word library.

[0011] In some embodiments, determining candidate sensitive words in the text to be processed based on the start character marks and end character marks of each sensitive word includes: matching the start character of a sensitive word in the text to be processed based on the start character of each sensitive word; when the start character of a sensitive word is detected in the text to be processed, continuously detecting the end character of the sensitive word in the text to be processed; and obtaining a candidate sensitive word in response to detecting the end character of the sensitive word.

[0012] In some embodiments, before calculating the hash value of the candidate sensitive word and matching the hash value of the candidate sensitive word with the sensitive word hash table, the method also includes: creating a first hash function and a second hash function; wherein the first hash function is used to calculate the hash value of the candidate sensitive word, and the second hash function is used to update the candidate sensitive word.

[0013] In some embodiments, after acquiring at least one sensitive word by loading the sensitive word dictionary, the method further includes: reloading the sensitive word dictionary to complete hot update of the sensitive word dictionary.

[0014] In some embodiments, after filtering the sensitive words in the text to be processed to generate the target text, the method further includes: verifying whether the target text contains sensitive words and returning a Boolean value; if the Boolean value is true, the target text contains sensitive words; if the Boolean value is false, the target text does not contain sensitive words.

[0015] According to another aspect of the present disclosure, a text processing device is also provided, the device comprising: an acquisition module, used to acquire a sensitive word library and a text to be processed; the sensitive word library comprises a sensitive word hash table, the start character and the end character of each sensitive word; a candidate sensitive word determination module, used to determine a candidate sensitive word in the text to be processed based on the start character and the end character of each sensitive word; a matching module, used to calculate the hash value of the candidate sensitive word, and match the hash value of the candidate sensitive word with the sensitive word hash table; a sensitive word determination module, used to obtain the sensitive word in the text to be processed when the hash value of the candidate sensitive word is successfully matched in the sensitive word hash table; a replacement module, used to replace the sensitive word in the text to be processed to generate a text without sensitive words.

[0016] According to another aspect of the present disclosure, an electronic device is also provided, which includes: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned text processing methods by executing the executable instructions.

[0017] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the text processing method described in any one of the above is implemented.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including: a computer program or instructions, wherein when the computer program or instructions are executed by a processor, any one of the above-mentioned text processing methods is implemented.

[0019] A text processing method, device and related equipment are provided in the embodiments of the present disclosure. The method includes: obtaining a text to be processed; determining a candidate sensitive word in the text to be processed based on the start character and end character of each sensitive word in the sensitive word library, wherein the sensitive word library includes a sensitive word hash table, the start character and the end character of each sensitive word; calculating the hash value of the candidate sensitive word, and matching the hash value of the candidate sensitive word with the sensitive word hash table; when the hash value of the candidate sensitive word is successfully matched in the sensitive word hash table, the sensitive word in the text to be processed is obtained; the sensitive word in the text to be processed is replaced to generate a text without sensitive words. The present disclosure first uses character position information to quickly locate possible candidate sensitive words in the text, and then calculates the hash value of the candidate sensitive word and matches it with the sensitive word hash table to further accelerate the recognition process of the sensitive word. This double-layer positioning and matching mechanism effectively reduces false positives and can improve the efficiency and accuracy of sensitive word filtering.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 A schematic diagram showing a system architecture of a text processing method in an embodiment of the present disclosure;

[0023] Figure 2 A method flow chart showing a text processing method in an embodiment of the present disclosure;

[0024] Figure 3 A flow chart of a method for generating a sensitive word library in an embodiment of the present disclosure is shown;

[0025] Figure 4 A flow chart of a method for placing a hash sensitive word list into a JAVA project in an embodiment of the present disclosure is shown;

[0026] Figure 5 A flow chart of a method for annotating sensitive words in an embodiment of the present disclosure is shown;

[0027] Figure 6 A flow chart of a method for determining candidate sensitive words in an embodiment of the present disclosure is shown;

[0028] Figure 7 A flow chart of a sensitive word verification method in an embodiment of the present disclosure is shown;

[0029] Figure 8 A flowchart of a specific implementation method of a text processing method shown in an embodiment of the present disclosure;

[0030] Fig. 9 A flowchart of a specific implementation method of a text processing method shown in an embodiment of the present disclosure;

[0031] Fig.10 A schematic diagram of a text processing device in an embodiment of the present disclosure is shown;

[0032] Fig.11 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0035] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0036] For ease of understanding, before introducing the embodiments of the present disclosure, several terms involved in the embodiments of the present disclosure are first explained as follows:

[0037] Sensitive word filtering is a content review technology used to identify and process words in text that are inappropriate, offensive, or violate specific policies. Sensitive word filtering technology is widely used in Internet platforms, social media, forums, chat applications, and other scenarios where user-generated content needs to be monitored and managed. The main purpose of sensitive word filtering is to maintain the health, safety, and compliance of the network environment and prevent inappropriate speech from having a negative impact on users and society.

[0038] A hash table, also known as a hash table, is a data structure that organizes data through a hash function. It provides fast data insertion, deletion, and search capabilities. The working principle of a hash table is based on converting the input (such as a key value) into an index of an array (or slot) through a hash function, which points to the location where the data is stored.

[0039] The TTMP algorithm (Two-Tier Matching with Positioning) is a two-tier positioning matching algorithm for text matching. The two-tier matching is divided into two stages: inner and outer. Inner: Preliminary screening of possible matching pairs. Outer: Further optimization and confirmation of matching results. Positioning: Introducing position information in the matching process to improve matching accuracy.

[0040] Text detection refers to the process of analyzing and processing the content of a text to be processed, with the goal of identifying, extracting or verifying specific information in the text. This process can be applied in a variety of scenarios, including but not limited to data mining, information retrieval, content review and natural language processing.

[0041] Algorithm efficiency refers to the resource consumption required by the algorithm to perform tasks, mainly including time complexity and space complexity. It is an important indicator to measure algorithm performance, especially in scenarios where large-scale data is processed or fast response is required.

[0042] The specific implementation of the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0043] Figure 1 FIG. 2 shows an exemplary application system architecture diagram to which the text processing method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0044] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0045] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0046] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, smart speakers, smart watches, wearable devices, augmented reality devices, virtual reality devices, etc.

[0047] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0048] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 101. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0049] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0050] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0051] Under the above system architecture, a text processing method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0052] In some embodiments, the text processing method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the text processing method provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the text processing method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0053] Figure 2 A flow chart of a text processing method in an embodiment of the present disclosure is shown as follows: Figure 2 As shown, the text processing method provided in the embodiment of the present disclosure includes the following steps:

[0054] S202, obtaining the text to be processed.

[0055] The text to be processed may include user input, file reading, network data, etc. The sources of the text to be processed may include social media, chat applications, email services, online education platforms, e-commerce platforms, and other fields.

[0056] S204: Determine candidate sensitive words in the text to be processed based on the start character and the end character of each sensitive word in the sensitive word library.

[0057] It should be noted that the sensitive word library may include a sensitive word hash table, the start character and the end character of each sensitive word. The start character refers to the first character of each sensitive word, and the end character refers to the last character of each sensitive word. The candidate sensitive word refers to a string fragment in the text to be processed that matches the start character and the end character recorded in the sensitive word library.

[0058] In this embodiment, the detection of sensitive words in the text to be processed is not to match sensitive words word by word, but to first locate the possible starting position according to the starting character of the sensitive word, and then determine its end point according to the ending character, and lock the "candidate sensitive word" in the text to be processed. In other words, if the beginning and ending characters of a certain text in the text to be processed match the corresponding characters of a word in the sensitive word library, then it is a possible candidate sensitive word. This method is highly efficient and suitable for processing large amounts of text data.

[0059] S206, calculating the hash value of the candidate sensitive word, and matching the hash value of the candidate sensitive word with the sensitive word hash table.

[0060] In this embodiment, a hash value is an algorithm result that converts data of any length into a fixed-length string through a hash function. Calculating the hash value of a candidate sensitive word means that each candidate sensitive word is converted into a unique hash value through a hash function. Hash values ​​can more efficiently store, retrieve, and compare sensitive words. A sensitive word hash table is a pre-established storage structure that stores known sensitive words and their corresponding hash values.

[0061] Matching the hash value of the candidate sensitive word with the sensitive word hash table is to find out whether the candidate sensitive word is a known sensitive word in the sensitive word hash table.

[0062] In some embodiments, before calculating the hash value of the candidate sensitive word and matching the hash value of the candidate sensitive word with the sensitive word hash table, the method further includes: creating a first hash function and a second hash function, wherein the first hash function is used to calculate the hash value of the candidate sensitive word, and the second hash function is used to update the candidate sensitive word.

[0063] Specifically, the first hash function may be hash(int hash, int len, String word). This function is used to calculate the hash value of the entire string (candidate sensitive word). Among them, the parameter "hash" is the current hash value, "len" is the length of the string, and "word" is the string for which the hash value is to be calculated.

[0064] The second hash function can be hash(int hash, Character ch). This function is used to update the hash value based on the current character. The parameter "hash" is the current hash value, and "ch" is the next character to be processed. It is used to process the string character by character, and only one character needs to be processed at a time, which can efficiently iterate and update the hash value.

[0065] In this embodiment, when designing a hash function, the hash function can be made to evenly distribute sensitive words in the hash table, thereby reducing conflicts and thus reducing the possibility of false positives and false negatives.

[0066] S208, when the hash value of the candidate sensitive word successfully matches in the sensitive word hash table, the sensitive word in the text to be processed is obtained.

[0067] If the match is successful, it means that the candidate sensitive word is a sensitive word; if the match fails, it means that the candidate sensitive word is not a sensitive word. This method can improve detection efficiency and reduce the time cost of one-by-one comparison.

[0068] S210, replacing sensitive words in the text to be processed to generate a text without sensitive words.

[0069] In this embodiment, for the detected sensitive words, the sensitive words in the to-be-processed text are replaced with a specified replacement value (such as * or -). It should be noted that the user can select the replacement value as needed, and this embodiment supports flexible replacement of sensitive words.

[0070] Specifically, this embodiment provides a replace(String text, Character repval) method to replace sensitive words in text with specified replacement values. "String text" refers to the original string to be replaced. "Character repval" refers to the new character used for replacement. This method will traverse each character in "text (text to be processed)", and when encountering a character that needs to be replaced, it will replace it with "repval". Finally, a new string is returned, in which all specified characters have been replaced. This method supports streaming text processing and can be applied in JAVA projects.

[0071] For example, replace sensitive words with "-". The code example is as follows:

[0072] String filteredText = filter.replace("This contains some sensitive words and normal text.",'-');

[0073] System.out.println("Filtered text:"+filteredText).

[0074] In this embodiment, based on the TTMP algorithm, the first layer first uses character position information to quickly locate possible candidate sensitive words in the text, and then the second layer calculates hash values ​​for the candidate sensitive words and matches them with the sensitive word hash table to further accelerate the recognition process of sensitive words. Through the double-layer matching and positioning method, false alarms are effectively reduced, which can improve the efficiency and accuracy of sensitive word filtering.

[0075] Figure 3 The following is a flow chart of a method for generating a sensitive word library as shown in the present disclosure. Figure 3 As shown in the figure, the sensitive word library is generated in the following way:

[0076] S302, placing the hash sensitive word list into the JAVA project, and annotating the sensitive words in the hash sensitive word list to generate a sensitive word library.

[0077] JAVA projects refer to software projects developed using the Java programming language. The hash sensitive word list is a database or list that stores sensitive words, including sensitive words and their corresponding hash values. Annotation labeling refers to additional explanations or descriptions of sensitive words, such as labeling the part of speech, severity, violation type, character position information, etc. of sensitive words. It is used to quickly understand the meaning and context of sensitive words, so as to make more accurate judgments and processing.

[0078] In this embodiment, a hash sensitive word list is placed in the JAVA project, that is, a list containing sensitive words is integrated into the program. Generating a sensitive word library is to organize the hash sensitive word list and the annotated sensitive words to form a structured database or file for the program to call at runtime.

[0079] Specifically, Map can be used <Integer,Set <string>>Data structure, with hash value as key and sensitive word set as value, to achieve fast retrieval and management of sensitive words.

[0080] In some embodiments, Figure 4 The following is a flow chart of a method for placing a hash sensitive word list into a JAVA project. Figure 4 As shown, placing the hash sensitive word list into the JAVA project can include:

[0081] S402, creating a sensitive word hash table in the JAVA project.

[0082] Among them, the key of the sensitive word hash table is used to store the hash value of the sensitive word, and the value of the sensitive word hash table is used to store the sensitive word.

[0083] S404: Obtain at least one sensitive word by loading a sensitive word dictionary.

[0084] In this embodiment, the sensitive word dictionary is a tool or database for storing and managing sensitive words, and can be loaded from a file or an input stream.

[0085] Specifically, it is implemented through the loadDictionary(File file), loadDictionary(String filename) and loadDictionary(InputStream input) methods.

[0086] Among them, loadDictionary(File file) receives a "File" object as a parameter, which represents a specific file. When calling this method, you need to pass in an existing file instance, and the method will read the contents of the file to load the dictionary.

[0087] loadDictionary(String filename) receives a string file name. This method will find and open the corresponding file according to the provided file name, and then read the content to load the dictionary.

[0088] loadDictionary(InputStreaminput) is more flexible, it receives an `InputStream` stream. This means that the dictionary can be loaded from any data source that can produce an input stream, such as a network stream, a memory stream, etc. This method provides the greatest flexibility, allowing dictionary data to be loaded without directly operating the file.

[0089] In some embodiments, the embodiments of the present disclosure allow users to add or delete words in the sensitive word library as needed, providing customization capabilities.

[0090] In some embodiments, after acquiring at least one sensitive word by loading the sensitive word dictionary, the method further includes: reloading the sensitive word dictionary to complete hot update of the sensitive word dictionary.

[0091] In this embodiment, the system is allowed to update the sensitive word library during operation to adapt to new sensitive word filtering requirements.

[0092] S406, calculating the hash value of each sensitive word, and storing each sensitive word and the hash value corresponding to each sensitive word in a sensitive word hash table.

[0093] In this embodiment, for sensitive words, key features (such as letters, numbers, etc.) are extracted, and a hash function is designed based on these features to improve the accuracy of matching. The calculation method of the hash value of the sensitive word can refer to the hash value calculation method of the candidate sensitive word recorded above, which will not be repeated here.

[0094] In some embodiments, when calculating the hash value of a sensitive word, the length of the sensitive word is considered to ensure that sensitive words of different lengths can be effectively processed. Specifically, for sensitive words of different lengths, different levels of hash functions are used to reduce conflicts between long sensitive words and short sensitive words.

[0095] The following is a specific implementation of the TTMP algorithm, showing how to optimize the hash function and handle hash conflicts:

[0096]

[0097]

[0098] In this embodiment, the performance of the TTMP algorithm is improved by optimizing the hash function and using the linked list method to handle hash conflicts. Through these optimization strategies, the TTMP algorithm can show better performance and accuracy in matching sensitive words of different lengths.

[0099] Specifically, each sensitive word and the hash value corresponding to each sensitive word are stored in the sensitive word hash table, and a single sensitive word can be added to the sensitive word library through the add(String word) method. Where add represents an add or insert operation. The String word parameter indicates that the add method accepts a string word string type data as input.

[0100] In some embodiments, in order to adapt to different load factors and maintain the performance of the hash table, the size of the hash table can be dynamically adjusted according to actual usage.

[0101] In some embodiments, for frequently accessed sensitive words, a cache mechanism is used to store the most recent or most frequently matched sensitive words to improve matching efficiency.

[0102] Figure 5 A flowchart of a method for annotating sensitive words shown in an embodiment of the present disclosure, combined with Figure 5 As shown, the sensitive words in the hash sensitive word list are annotated to generate a sensitive word library, which may include:

[0103] S502: Create a first character tag array and a second character tag array.

[0104] The first character marker array is used to mark the start characters of each sensitive word, and the second character marker array is used to mark the end characters of each sensitive word.

[0105] Specifically, two Boolean arrays boolean[]first and boolean[]end can be created. A Boolean array is a special array whose elements contain only two possible values: true (usually represented as True) and false (usually represented as False). The length of a Boolean array can be arbitrary, and the element at each position represents a Boolean value. Boolean arrays can be used to quickly determine whether an element in a list meets a certain condition, or to indicate the existence or non-existence of an element in a set. In this embodiment, the Boolean array boolean[]first and boolean[]end arrays respectively mark the start and end characters of sensitive words.

[0106] S504: Update the first character mark array and the second character mark array of each sensitive word to mark the start character and the end character of each sensitive word.

[0107] In this embodiment, sensitive words are read from a file or input stream, a hash value is calculated for each sensitive word, and the result is stored in a hash table. At the same time, the first and end arrays are updated to mark the first character (start character) and the last character (end character) of each sensitive word.

[0108] S506: Generate a sensitive word library based on the sensitive word hash table, the updated first character tag array of the start character of each sensitive word, and the second character tag array of the end character.

[0109] In some embodiments, Figure 6 The following is a flow chart of a method for determining candidate sensitive words shown in the present disclosure. Figure 6 As shown, based on the start character mark and the end character mark of each sensitive word, candidate sensitive words are determined in the text to be processed, including:

[0110] S602: Based on the start characters of each sensitive word, the start characters of the sensitive words are matched in the text to be processed.

[0111] S604: When the start character of a sensitive word is detected in the text to be processed, the end character of the sensitive word is continuously detected in the text to be processed.

[0112] In this embodiment, once the start character of a sensitive word is detected in the text to be processed, continuous detection will continue along the text at the position of the start character to determine whether subsequent characters constitute a continuation of the sensitive word until the end character is encountered.

[0113] S606: In response to detecting the end character of the sensitive word, a candidate sensitive word is obtained.

[0114] In this embodiment, only the start character and the end character of the sensitive word are initially matched, which helps to quickly identify candidate sensitive words in the text.

[0115] In some embodiments, Figure 7 A flow chart of a sensitive word verification method provided by an embodiment of the present disclosure. Figure 7 As shown, after filtering the sensitive words in the to-be-processed text to generate the target text, the method further includes:

[0116] S702, verify whether the target text contains sensitive words and return a Boolean value.

[0117] It should be noted that Boolean values ​​are derived from the logical algebra created by mathematician Boole, and have only two states: true (usually represented as True) and false (usually represented as False).

[0118] In this embodiment, the verification method is used to verify whether a given text contains any sensitive words and return a Boolean value. The verification method is used to confirm whether something meets a specific standard or condition. In this embodiment, the verification method is used to verify whether the target text contains sensitive words.

[0119] Specifically, a sensitive word library is first established, covering all kinds of sensitive words that need to be filtered; then the input target text is scanned and compared one by one to see if it matches the words in the sensitive word library; if a match is found, the text is determined to contain sensitive words and a Boolean value of "True" is returned; if not found, "False" is returned.

[0120] The code example is as follows:

[0121] boolean containsSensitive = filter.verification("This contains some sensitive words and normal text.

[0122] ");

[0123] System.out.println("Contains sensitive words:"+containsSensitive).

[0124] S704, if the Boolean value is true, the target text contains sensitive words.

[0125] S706, if the Boolean value is false, the target text does not contain sensitive words.

[0126] Figure 8 The following is a flowchart of a specific implementation method of a text processing method shown in an embodiment of the present disclosure. Figure 8 As shown, the text processing method may include:

[0127] S801, initialize the WordFilter object.

[0128] In this embodiment, a sensitive word filter object is created using the WordFilter class. This object will be used to load the sensitive word library, detect and replace sensitive words. The code example is as follows: WordFilter filter = new WordFilter ("path / to / sensitive_words.dic").

[0129] In some embodiments, a WordFilterTest class may also be created to provide test cases to demonstrate how to use the WordFilter class to detect and replace sensitive words.

[0130] S802, whether to load the dictionary, if yes, go to S803, if not, go to S804.

[0131] S803, loading a dictionary from a file or an input stream.

[0132] In this embodiment, the sensitive word library is loaded from the specified file or resource. This step is completed in the constructor, and can also be explicitly called through the loadDictionary method. For example, filter.loadDictionary(new File("path / to / sensitive_words.dic")).

[0133] S804, continue execution.

[0134] S805, adding sensitive words to the hash table.

[0135] S806, receiving the text to be detected.

[0136] S807, detect sensitive words, if sensitive words are found, go to S808, if sensitive words are not found, go to step S809.

[0137] In this embodiment, the judge method can be used to implement sensitive word detection and return a set of all sensitive words. The judge method is a function or process used to determine whether a given input meets a specific condition or satisfies a certain logical requirement. It receives the text to be processed as input, passes through a series of pre-set judgment logic, and then outputs the judgment result.

[0138] Specifically, the judge method is implemented using the enhanced for loop in the Java language. The following is a possible code example provided in this embodiment:

[0139] Set <string>sensitiveWords = filter.judge("This contains some sensitive words and normal text.");

[0140] for(String word:sensitiveWords){

[0141] System.out.println("Detected sensitive word:"+word);

[0142] }

[0143] In this embodiment, the provided judge(String text) method can be used to search and return all sensitive words in the text, and supports deduplication.

[0144] S808, replace sensitive words.

[0145] S809, output the original text.

[0146] S810, output the replaced text.

[0147] In this embodiment, the system is developed based on JAVA, has good cross-platform compatibility, and can run on a variety of operating systems.

[0148] Fig. 9 The present invention is a flowchart of a specific implementation method of a text processing method shown in an embodiment of the present invention.

[0149] S901, initialize the hash table and array.

[0150] In this embodiment, a hash table is created to store sensitive words, the key is the hash value of the sensitive word, and the value is the sensitive word itself. Two Boolean arrays first and end are initialized to mark the start and end characters of the sensitive word respectively.

[0151] In some embodiments, a hash table is a data structure that maps keys to slots through a hash function. However, different keys may have the same hash value, which results in a hash conflict. The TTMP algorithm can use the following linked list method, open addressing method, and double hashing to handle hash conflicts.

[0152] The linked list method uses a linked list to store sensitive words with the same hash value in each slot of the hash table. When a hash conflict is detected, the algorithm traverses the linked list of the slot and determines whether it is a sensitive word through precise string matching.

[0153] The open addressing method is to find another free position in the table to store the current sensitive word. This method requires a detection sequence, such as linear detection or quadratic detection.

[0154] Double hashing uses two hash functions, and when the first hash function produces a conflict, the second hash function is used to calculate the new position. This method can reduce the probability of conflict.

[0155] S902, loading a sensitive word library.

[0156] In this embodiment, sensitive words are read from a file or an input stream, a hash value is calculated for each sensitive word, and the hash value is stored in a hash table, and the first and end arrays are updated at the same time to mark the first and last characters of each sensitive word.

[0157] S903, traverse the text to be detected, if there is a start character, go to S904, if there is no start character, go to S910.

[0158] S904, detect the start character, if yes, go to S905, if not, go to S903.

[0159] S905, construct candidate words.

[0160] S906, detect the end character, if yes, go to S907, if not, go to S905.

[0161] S907, calculating the hash value of the candidate word.

[0162] S908, determine whether the hash value matches, if so, go to S909, if not, go to S903.

[0163] S909, replace sensitive words.

[0164] S910, output result.

[0165] In this embodiment, the TTMP algorithm is adopted, and the efficiency and accuracy of sensitive word matching are improved by combining the hash table and the hash function.

[0166] In some embodiments, when processing a large amount of text to be processed, parallel processing technology is used to divide the data into multiple small blocks, and the TTMP algorithm is run on multiple processors at the same time to improve the overall performance.

[0167] Based on the same inventive concept, the present disclosure also provides a text processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0168] Fig.10 A schematic diagram of a text processing device in an embodiment of the present disclosure is shown. Fig.10 As shown, the device includes: an acquisition module 1011, a candidate sensitive word determination module 1012, a matching module 1013, a sensitive word determination module 1014 and a replacement module 1015.

[0169] The acquisition module 1011 is used to acquire a sensitive word library and a text to be processed; the sensitive word library includes a sensitive word hash table, and the start character and end character of each sensitive word;

[0170] A candidate sensitive word determination module 1012 is used to determine candidate sensitive words in the text to be processed based on the start character and the end character of each sensitive word;

[0171] The matching module 1013 is used to calculate the hash value of the candidate sensitive word and match the hash value of the candidate sensitive word with the sensitive word hash table;

[0172] A sensitive word determination module 1014 is used to obtain the sensitive words in the text to be processed when the hash value of the candidate sensitive word successfully matches in the sensitive word hash table;

[0173] The replacement module 1015 is used to replace the sensitive words in the text to be processed to generate a text without sensitive words.

[0174] In some embodiments, the acquisition module 1011 is further used to: place the hash sensitive word list into the JAVA project, and annotate the sensitive words in the hash sensitive word list to generate a sensitive word library.

[0175] In some embodiments, the acquisition module 1011 is also used to: create a sensitive word hash table in a JAVA project; wherein the key of the sensitive word hash table is used to store the hash value of the sensitive word, and the value of the sensitive word hash table is used to store the sensitive word; obtain at least one sensitive word by loading a sensitive word dictionary; calculate the hash value of each sensitive word, and store each sensitive word and the hash value corresponding to each sensitive word in the sensitive word hash table.

[0176] In some embodiments, the acquisition module 1011 is also used to: create a first character marker array and a second character marker array; wherein the first character marker array is used to mark the start character of each sensitive word, and the second character marker array is used to mark the end character of each sensitive word; update the first character marker array and the second character marker array of each sensitive word to mark the start character and the end character of each sensitive word; generate a sensitive word library based on the sensitive word hash table, the updated first character marker array of the start character of each sensitive word and the second character marker array of the end character.

[0177] In some embodiments, the candidate sensitive word determination module 1012 is specifically used to: match the start characters of sensitive words in the text to be processed based on the start characters of each sensitive word; when the start characters of the sensitive words are detected in the text to be processed, continuously detect the end characters of the sensitive words in the text to be processed; in response to detecting the end characters of the sensitive words, obtain candidate sensitive words.

[0178] In some embodiments, the candidate sensitive word determination module 1012 is also used to: create a first hash function and a second hash function; wherein the first hash function is used to calculate the hash value of the candidate sensitive word, and the second hash function is used to update the candidate sensitive word.

[0179] In some embodiments, the acquisition module 1011 is further used to: reload the sensitive word dictionary to complete the hot update of the sensitive word dictionary.

[0180] In some embodiments, the device also includes a verification module 1015, which is used to: verify whether the target text contains sensitive words and return a Boolean value; if the Boolean value is true, the target text contains sensitive words; if the Boolean value is false, the target text does not contain sensitive words.

[0181] It should be noted that the examples and application scenarios implemented by the modules in the above-mentioned device embodiment are the same as those of the corresponding steps in the method embodiment, but are not limited to the contents disclosed in the above-mentioned method embodiment. It should be noted that the above-mentioned modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0182] Those skilled in the art will appreciate that various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit", "module" or "system".

[0183] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned text processing methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above-mentioned method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0184] Refer to the following Fig.11 1100 according to this embodiment of the present disclosure is described. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0185] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include but are not limited to: at least one processing unit 1110, at least one storage unit 1120, and a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110).

[0186] The storage unit stores a program code, which can be executed by the processing unit 1110, so that the processing unit 1110 executes the steps described in the "Exemplary Method" section of the specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1110 can execute the following steps of the above method embodiment: obtain the text to be processed; determine the candidate sensitive words in the text to be processed based on the start character and end character of each sensitive word in the sensitive word library, wherein the sensitive word library includes a sensitive word hash table, the start character and the end character of each sensitive word; calculate the hash value of the candidate sensitive word, and match the hash value of the candidate sensitive word with the sensitive word hash table; when the hash value of the candidate sensitive word is successfully matched in the sensitive word hash table, the sensitive word in the text to be processed is obtained; replace the sensitive words in the text to be processed to generate a text without sensitive words.

[0187] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 11201 and / or a cache storage unit 11202 , and may further include a read-only storage unit (ROM) 11203 .

[0188] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0189] Bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0190] The electronic device 1100 may also communicate with one or more external devices 1140 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0191] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0192] Based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned text processing methods is implemented. Since the principle of solving the problem in the embodiment of the computer-readable storage medium is similar to that in the above-mentioned method embodiment, the implementation of the embodiment of the computer-readable storage medium can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0193] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0194] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0195] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0196] In specific implementations, program codes for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0197] Based on the same inventive concept, a computer program product is also provided in the embodiments of the present disclosure, including a computer program product, including: a computer program or an instruction, wherein when the computer program or the instruction is executed by a processor, the text processing method of any one of the above method embodiments is implemented. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0198] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0199] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0200] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0201] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.< / string> < / string>

Claims

1. A text processing method, characterized in that: The method comprises: Get the text to be processed; Determining candidate sensitive words in the to-be-processed text based on the start character and the end character of each sensitive word in the sensitive word library, wherein the sensitive word library includes a sensitive word hash table, the start character and the end character of each sensitive word; Calculating the hash value of the candidate sensitive word, and matching the hash value of the candidate sensitive word with the sensitive word hash table; When the hash value of the candidate sensitive word successfully matches in the sensitive word hash table, the sensitive word in the to-be-processed text is obtained; The sensitive words in the text to be processed are replaced to generate a text without sensitive words.

2. The text processing method according to claim 1, characterized in that: The sensitive word library is generated in the following way: Place the hash sensitive word list into the JAVA project, annotate the sensitive words in the hash sensitive word list, and generate a sensitive word library.

3. The text processing method according to claim 2, characterized in that: Place the hash sensitive word list into the JAVA project, including: Create a sensitive word hash table in a JAVA project; wherein the key of the sensitive word hash table is used to store the hash value of the sensitive word, and the value of the sensitive word hash table is used to store the sensitive word; Obtain at least one sensitive word by loading the sensitive word dictionary; Calculate the hash value of each sensitive word, and store each sensitive word and the hash value corresponding to each sensitive word in the sensitive word hash table.

4. The text processing method according to claim 2, characterized in that: Annotate the sensitive words in the hash sensitive word list to generate a sensitive word library, including: Create a first character marker array and a second character marker array; wherein the first character marker array is used to mark the start characters of each sensitive word, and the second character marker array is used to mark the end characters of each sensitive word; Update the first character mark array and the second character mark array of each sensitive word to mark the start character and the end character of each sensitive word; A sensitive word library is generated based on the sensitive word hash table, the updated first character tag array of the start character of each sensitive word, and the second character tag array of the end character.

5. The text processing method according to claim 1, characterized in that: The step of determining candidate sensitive words in the text to be processed based on the start character mark and the end character mark of each sensitive word includes: Based on the starting characters of each sensitive word, matching the starting characters of the sensitive words in the text to be processed; When the start character of a sensitive word is detected in the text to be processed, the end character of the sensitive word is continuously detected in the text to be processed; In response to detecting the end character of the sensitive word, a candidate sensitive word is obtained.

6. The text processing method according to claim 1, characterized in that: Before calculating the hash value of the candidate sensitive word and matching the hash value of the candidate sensitive word with the sensitive word hash table, the method further includes: Create a first Hash function and a second Hash function; wherein the first Hash function is used to calculate the hash value of the candidate sensitive word, and the second Hash function is used to update the candidate sensitive word.

7. The text processing method according to claim 1, characterized in that: After obtaining at least one sensitive word by loading the sensitive word dictionary, it also includes: Reload the sensitive word dictionary to complete the hot update of the sensitive word dictionary.

8. The text processing method according to claim 1, characterized in that: After filtering the sensitive words in the text to be processed to generate the target text, the method further includes: Verify whether the target text contains sensitive words and return a Boolean value; If the Boolean value is true, the target text contains sensitive words; If the Boolean value is false, the target text does not contain sensitive words.

9. A text processing device, characterized in that: The device comprises: An acquisition module is used to acquire a sensitive word library and a text to be processed; the sensitive word library includes a sensitive word hash table, and the start character and end character of each sensitive word; A candidate sensitive word determination module, used to determine candidate sensitive words in the text to be processed based on the start character and the end character of each sensitive word; A matching module, used to calculate the hash value of the candidate sensitive word, and match the hash value of the candidate sensitive word with the sensitive word hash table; A sensitive word determination module, configured to obtain the sensitive words in the text to be processed when the hash value of the candidate sensitive word successfully matches the sensitive word hash table; The replacement module is used to replace the sensitive words in the text to be processed to generate a text without sensitive words.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to perform the text processing method described in any one of claims 1 to 8 by executing the executable instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the text processing method described in any one of claims 1 to 8.

Images