Sensitive word storage and retrieval method and system, electronic equipment and storage medium

Through the target mapping table and sharding service, sensitive words are stored and retrieved, which solves the problem of low storage and retrieval efficiency of sensitive words in the prior art, and realizes efficient sensitive word retrieval and similar word detection, reducing costs.

CN118916543AActive Publication Date: 2024-11-08广域铭岛数字科技有限公司 +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410958281.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-17
Publication Date
2024-11-08
Estimated Expiration
2044-07-17

AI Technical Summary

Technical Problem

In the prior art, sensitive words storage and retrieval efficiency are low, and detection of similar sensitive words cannot be effectively realized.

Method used

The target mapping table stores the sensitive words to be stored in the target shard service, and determines the shard service to be retrieved based on the target mapping table and the first character of the word to be retrieved. The sensitive words are retrieved using Trie or dictionary data structure to realize shard storage and efficient search.

Benefits of technology

It improves the storage and retrieval efficiency of sensitive words, reduces the load pressure of sharded services, saves memory space, and realizes efficient retrieval of massive sensitive words.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916543A_ABST
    Figure CN118916543A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, in particular to a sensitive word storage and retrieval method and system, electronic equipment and a storage medium, the method comprises the steps that a to-be-stored sensitive word is stored to a target fragmentation service through a target mapping table, the target mapping table comprises a mapping relation between the target fragmentation service and a first character of the to-be-stored sensitive word, and the first character of the to-be-stored sensitive word is stored in the target fragmentation service; the target fragmentation service is one of a plurality of fragmentation services of a memory, and the target fragmentation service is the fragmentation service polled at present; based on the target mapping table and the first character of the to-be-retrieved word, determining a to-be-retrieved fragment service corresponding to the first character of the to-be-retrieved word; based on the to-be-retrieved words and data stored in the to-be-retrieved fragmentation service, sensitive word retrieval is carried out; according to the method, the storage efficiency and retrieval efficiency of the sensitive words can be well improved, and the cost is low.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a sensitive word storage and retrieval method, system, electronic device, and storage medium. Background Art

[0002] Sensitive word search is crucial in modern social media and online platforms. It can help effectively prevent inappropriate content from appearing in information dissemination, to ensure the safety and civility of online communication, as well as the compliance and legality of information dissemination. Currently, some sensitive words are usually built into (stored) the database. When sensitive word search is needed, the sensitive words built into the database are matched to complete the sensitive word search.

[0003] However, the above method has the following problems: 1. When using the database to store and retrieve sensitive words, it is necessary to access the database and traverse information, and due to the large number of sensitive words, the efficiency of sensitive word storage and retrieval is low; 2. The above method cannot effectively detect similar sensitive words. Summary of the invention

[0004] The present invention provides a sensitive word storage and retrieval method, system, electronic device, and storage medium to solve the problem of low sensitive word storage and retrieval efficiency in the prior art.

[0005] The present invention provides a sensitive word storage and retrieval method, the method comprising:

[0006] The sensitive words to be stored are stored in the target sharding service through the target mapping table, wherein the target mapping table includes a mapping relationship between the target sharding service and the first character of the sensitive words to be stored, the target sharding service is one of the multiple sharding services in the memory, and the target sharding service is the sharding service currently polled;

[0007] Based on the target mapping table and the first character of the word to be searched, a to-be-searched slice service corresponding to the first character of the word to be searched is determined; based on the word to be searched and the data stored in the to-be-searched slice service, a sensitive word search is performed.

[0008] In one embodiment of the present invention, the sensitive word to be stored is the newly added sensitive word itself or a synonym of the newly added sensitive word; if the sensitive word is stored for the first time, the target shard service is the first shard service among multiple shard services; the mapping relationship between the target shard service and the first character of the sensitive word to be stored is established before the sensitive word to be stored is stored in the target shard service, and after the mapping relationship between the target shard service and the first character of the sensitive word to be stored is established, it also includes:

[0009] The preset polling shard number is updated to the shard number of the next shard service of the current target shard service, and the corresponding target shard service when sensitive words are stored next time is determined based on the polling shard number.

[0010] In one embodiment of the present invention, the step of storing the sensitive words to be stored in the target shard service through the target mapping table includes:

[0011] Determining the target sharding service based on the first character of the sensitive word to be stored and the target mapping table;

[0012] Obtain the target address of the target shard service;

[0013] Sending a storage request to the target address, the storage request including the sensitive word to be stored, the storage request being used to instruct the target sharding service to perform character splitting on the sensitive word to be stored to obtain at least two target characters;

[0014] When the preset storage requirement is single data structure storage, the target characters are stored according to the Trie data structure or the dictionary data structure; when the storage requirement is redundant data structure storage, the target characters are stored according to the Trie data structure and the dictionary data structure respectively.

[0015] In one embodiment of the present invention, when the preset storage requirement is single data structure storage, the step of storing the target characters according to the Trie data structure or the dictionary data structure includes:

[0016] When the preset storage requirement is single data structure storage, the target character is stored according to the Trie data structure;

[0017] Get the preset query conditions;

[0018] In the case where the query condition requires a full match search when performing a sensitive word search, the number of sensitive words and the number of characters of the sensitive words currently stored in the shard service are obtained, wherein the full match search refers to directly matching the words to be searched with the stored sensitive words; based on the number of sensitive words and the number of characters of sensitive words, a first memory occupancy size of all sensitive words currently stored in the shard service under the Trie data structure and a second memory occupancy size under the dictionary data structure are obtained;

[0019] If the second memory occupancy size is smaller than the first memory occupancy size, the data in the Trie data structure is converted into a dictionary data structure.

[0020] In one embodiment of the present invention, the step of obtaining the to-be-searched words includes:

[0021] Get the text to be retrieved;

[0022] Using a preset filter, the text to be searched is filtered to obtain a target text;

[0023] Using a preset word segmenter, segmenting the target text to obtain at least one word segment;

[0024] Performing synonym conversion on the participle to obtain a target synonym;

[0025] Matching the target synonym with historical sensitive words pre-stored in the cache, wherein the historical sensitive words are sensitive words that were successfully retrieved previously, and the successful retrieval means that sensitive words that are the same or similar to the previous target synonym are retrieved;

[0026] If the match is successful, the target synonym is determined to be a sensitive word; if the match fails, the target synonym is determined to be the word to be searched.

[0027] In one embodiment of the present invention, the step of performing sensitive word search based on the to-be-searched word and the data stored in the to-be-searched shard service includes:

[0028] Obtaining a data structure of the data stored in the shard service to be retrieved;

[0029] If the data structure of the data stored in the to-be-searched shard service is a Trie data structure, the to-be-searched word is split into characters to obtain at least two split characters; based on the split characters and the data stored in the to-be-searched shard service, a similarity judgment is performed to obtain the similarity between the to-be-searched word and the sensitive word stored in the to-be-searched shard service; if the similarity is greater than or equal to a preset similarity threshold, it is determined that the to-be-searched word is a sensitive word, and the to-be-searched word is stored in a cache;

[0030] If the data structure of the data stored in the to-be-searched shard service is a dictionary data structure, the to-be-searched word is matched with the sensitive words under the dictionary data structure; if the match is successful, the to-be-searched word is determined to be a sensitive word, and the to-be-searched word is stored in the cache.

[0031] In one embodiment of the present invention, it further includes:

[0032] If the first character of the current word to be searched is missing in the target mapping table, the current word to be searched is determined to be a normal word, and the search is stopped or the first character of the next word to be searched is matched with the first character in the target mapping table.

[0033] The present invention also provides a sensitive word storage and retrieval system, the system comprising: a sensitive word storage module, used to store the sensitive words to be stored in a target sharding service through a target mapping table, wherein the target mapping table comprises a mapping relationship between the target sharding service and the first character of the sensitive words to be stored, the target sharding service is one of the multiple sharding services in the memory, and the target sharding service is the sharding service currently polled;

[0034] The sensitive word search module is used to determine the to-be-searched fragment service corresponding to the first character of the to-be-searched word based on the target mapping table and the first character of the to-be-searched word; and perform sensitive word search based on the to-be-searched word and the data stored in the to-be-searched fragment service.

[0035] The present invention also provides an electronic device, comprising a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement a sensitive word storage and retrieval method as provided in any one of the above embodiments.

[0036] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to enable a computer to execute the sensitive word storage and retrieval method provided in any of the above embodiments.

[0037] Beneficial effects of the present invention: The sensitive word storage and retrieval method, system, electronic device, and storage medium proposed by the present invention store the sensitive words to be stored in the target sharding service through a target mapping table, wherein the target mapping table includes a mapping relationship between the target sharding service and the first character of the sensitive words to be stored, the target sharding service is one of the multiple sharding services in the memory, and the target sharding service is the sharding service currently polled; based on the target mapping table and the first character of the word to be retrieved, the sharding service to be retrieved corresponding to the first character of the word to be retrieved is determined; based on the word to be retrieved and the data stored in the sharding service to be retrieved, sensitive word retrieval is performed. The method stores all sensitive words in the memory, which can improve the storage efficiency better than storing them in a database. And by setting the target mapping table, it can quickly determine which sharding service the current word to be retrieved should be routed to, thereby realizing sensitive word retrieval, and the retrieval efficiency is high. Compared with making routing decisions through the HASH algorithm, the advantage of the above-mentioned method of setting the target mapping table is that the target mapping table takes the first character as the key instead of taking the entire vocabulary as the key, which can save memory space better. During the sensitive word search process, if the first character of the search word does not appear in the target mapping table, the search word can be considered a normal word, that is, a non-sensitive word. This method can complete the judgment and feedback of a large part of the search request in advance, while reducing the load pressure on the shard service and improving the search efficiency of mass sensitive word searches at a low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of a process flow of a sensitive word storage and retrieval method provided by an embodiment of the present invention;

[0039] Figure 2 A flowchart of a specific embodiment of storing (adding) sensitive words in the sensitive word storage and retrieval method provided in one embodiment of the present invention;

[0040] Figure 3 A flowchart of a specific embodiment of performing sensitive word retrieval in the sensitive word storage and retrieval method provided in one embodiment of the present invention;

[0041] Figure 4 It is a structural diagram of a sensitive word storage and retrieval system provided by an embodiment of the present invention;

[0042] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0045] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0046] The sensitive word storage and retrieval method, system, electronic device, and storage medium provided by the present invention are explained below by way of embodiments.

[0047] See also Figure 1 , Figure 1 A flow chart of a sensitive word storage and retrieval method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the sensitive word storage and retrieval method includes:

[0048] S110: The sensitive words to be stored are stored in the target sharding service through a target mapping table, wherein the target mapping table includes a mapping relationship between the target sharding service and the first characters of the sensitive words to be stored, the target sharding service is one of the multiple sharding services in the memory, and the target sharding service is the sharding service currently polled.

[0049] Specifically, when storing sensitive words for the first time, a target mapping table is established. In the target mapping table, a mapping relationship between the target slice service and the first character of the current sensitive word to be stored is established. At this time, the target slice service is the first slice service among the multiple slice services in the memory. Based on the mapping relationship in the target mapping table, the sensitive words to be stored are stored in the target slice service. Then, when storing the next sensitive word to be stored, the next slice service is polled to obtain the current target slice service. And in the existing target mapping table, a mapping relationship between the target slice service and the first character of the current sensitive word to be stored is established. Based on the mapping relationship, the current sensitive word to be stored is stored in the corresponding target slice service.

[0050] It should be noted that by establishing a target mapping table, i.e., a first character-slice number mapping table, it is possible to implement slicing storage of a large number of sensitive words, that is, to store a large number of sensitive words in multiple slicing services of the memory. For example: assuming that there are currently 900,000 sensitive words that need to be stored, and the memory is divided into 3 slicing services, then 300,000 of the sensitive words can be stored in the first slicing service, another 300,000 sensitive words can be stored in the second slicing service, and finally the remaining 300,000 sensitive words can be stored in the third slicing service. In addition, before storing the sensitive words in the corresponding slicing service, it is also necessary to first establish a mapping relationship between the first character of the sensitive word and the corresponding slicing service in the target mapping table, i.e., a mapping relationship between the first character of the sensitive word and the slicing number of the corresponding slicing service, so as to facilitate routing the sensitive words to the corresponding slicing service for storage. In addition, by establishing the target mapping table, it is also possible to facilitate the subsequent determination of the corresponding slicing service to be retrieved based on the target mapping table when performing sensitive word retrieval.

[0051] It should also be noted that when storing sensitive words, first, the sensitive words to be stored are obtained; secondly, if it is the first time to store sensitive words, a target mapping table is established, and a mapping relationship between the first character of the current sensitive words to be stored and the shard number of the first shard service is established in the target mapping table. At the same time, the preset polling shard number is updated, that is, updated to the shard number of the second shard service; then, based on the mapping relationship, the address of the first shard service is obtained, so that the current sensitive words to be stored are stored in the first shard service; thereafter, the second sensitive words to be stored are stored, the first character of the second sensitive words to be stored is obtained, and the current value of the polling shard number is obtained, and then the mapping relationship between the first character of the second sensitive words to be stored and the shard number of the second shard service is established, and then the second sensitive words to be stored are stored based on the mapping relationship. Repeat the above steps to achieve shard storage of massive sensitive words, with high storage efficiency.

[0052] It should be mentioned that, compared with storing sensitive words in a database, this embodiment can speed up storage and improve storage efficiency by storing the target mapping table and sensitive words in memory. By dividing the memory into multiple shard services, it is possible to achieve shard storage of massive sensitive words, which helps to improve the retrieval efficiency of sensitive words, and helps to achieve load balancing of memory and improve response speed.

[0053] S120: Based on the target mapping table and the first character of the word to be searched, determine the to-be-searched fragment service corresponding to the first character of the word to be searched; and perform sensitive word search based on the word to be searched and the data stored in the to-be-searched fragment service.

[0054] Specifically, in the process of sensitive word retrieval, based on the target mapping table and the first character of the word to be retrieved, the to-be-retrieved slice service corresponding to the first character of the word to be retrieved is determined; based on the word to be retrieved and the data stored in the to-be-retrieved slice service, sensitive word retrieval is performed.

[0055] It should be noted that, based on the target mapping table and the first character of the word to be searched, the step of determining the shard service to be searched corresponding to the first character of the word to be searched includes: matching the first character of the word to be searched with the first character in the target mapping table, if the match is successful, obtaining the shard number of the shard service corresponding to the first character in the target mapping table, and determining the shard service as the shard service to be searched. It can be understood that the shard service to be searched is the shard service corresponding to the first character of the word to be searched in the target mapping table. If the match fails, it is determined that the current word to be searched is a normal word, that is, a non-sensitive word, so that the retrieval and processing of the word to be searched are completed in advance, which greatly reduces the retrieval pressure and load pressure of the shard service, and effectively improves the retrieval efficiency of massive sensitive words.

[0056] In some embodiments, the sensitive word to be stored is the newly added sensitive word itself or a synonym of the newly added sensitive word. If the sensitive word storage is performed for the first time, the target shard service is the first shard service among the multiple shard services.

[0057] It should be noted that the newly added sensitive words refer to the sensitive words that need to be added and stored. By determining the newly added sensitive words themselves or the synonyms of the newly added sensitive words as the sensitive words to be stored, it can help improve the hit rate of subsequent sensitive word retrieval.

[0058] It should also be noted that if sensitive words are stored for the first time, the target shard service is the first shard service among multiple shard services; if sensitive words are stored for the second time, the target shard service is the second shard service among multiple shard services, and so on. By polling the shard number of the shard service, the corresponding target shard service can be determined.

[0059] In some embodiments, the mapping relationship between the target slicing service and the first character of the sensitive word to be stored is established before the sensitive word to be stored is stored in the target slicing service. After the mapping relationship between the target slicing service and the first character of the sensitive word to be stored is established, the method further includes:

[0060] The preset polling shard number is updated to the shard number of the next shard service of the current target shard service, and the corresponding target shard service when sensitive words are stored next time is determined based on the polling shard number.

[0061] It should be noted that, by updating the polling shard number to the shard number of the next shard service of the current target shard service after establishing a mapping relationship between the target shard service and the first character of the sensitive word to be stored, it is convenient to directly obtain the value of the polling shard number when storing sensitive words next time, find the corresponding shard service based on the polling shard number, determine the shard service as the target shard service at this time, and establish a mapping relationship between the current sensitive word to be stored and the target shard service.

[0062] In some embodiments, the step of storing the sensitive words to be stored in the target shard service through the target mapping table includes:

[0063] 1. Determine the target shard service based on the first character of the sensitive word to be stored and the target mapping table.

[0064] 2. Obtain the target address of the target shard service.

[0065] 3. Send a storage request to the target address, wherein the storage request includes the sensitive word to be stored, and the storage request is used to instruct the target sharding service to perform character splitting on the sensitive word to be stored to obtain at least two target characters.

[0066] 4. When the preset storage requirement is single data structure storage, the target characters are stored according to the Trie data structure or the dictionary data structure; when the storage requirement is redundant data structure storage, the target characters are stored according to the Trie data structure and the dictionary data structure respectively.

[0067] It should be noted that, through the above steps, different storage requirements can be met, namely, the requirements of single data structure storage and redundant data structure storage.

[0068] It should also be noted that, regarding the Trie (dictionary tree / prefix tree) data structure, Trie is a multi-branch tree, each node represents a character, and the path from the root node to each node constitutes a string. Storing the target characters according to the Trie data structure means building a string in the order of the target characters of the sensitive words to be stored to complete the storage of the sensitive words to be stored. By storing the target characters according to the Trie data structure, it is convenient to perform subsequent sensitive word retrieval and query. In addition, by storing the target characters according to the Trie data structure, it is also convenient to determine the similarity between the words to be retrieved and each sensitive word in the shard service to be retrieved based on the matching ratio between the words to be retrieved and each node in the Trie data structure in the subsequent sensitive word retrieval process, thereby realizing the detection of similar sensitive words. For example: Assuming that the current search term includes the four characters A, B, C, and D, these four characters are matched with the data in the Trie data structure in the shard service to be searched, that is, they are compared one by one in order. Assuming that the three characters A, B, and C are matched successfully, the matching ratio is 3 / 4, which is the similarity between the current search term and the sensitive word.

[0069] It should be mentioned that by storing the target characters according to the dictionary data structure, it is convenient to directly match the to-be-searched words with the sensitive words stored in the to-be-searched shard service during the subsequent sensitive word search process, which is more convenient to implement.

[0070] In some embodiments, when the preset storage requirement is single data structure storage, the step of storing the target characters according to the Trie data structure or the dictionary data structure includes:

[0071] 1. When the preset storage requirement is single data structure storage, the target character is stored according to the Trie data structure.

[0072] It should be noted that, since the Trie data structure can realize similarity retrieval between the searched word and the sensitive word, in this embodiment, when the storage requirement is single data structure storage, the target characters are preferentially stored according to the Trie data structure.

[0073] 2. Get the preset query conditions.

[0074] 3. When the query condition requires a full match search when searching for sensitive words, the number of sensitive words and the number of characters of the sensitive words currently stored in the shard service are obtained, and the full match search refers to directly matching the words to be searched with the stored sensitive words; based on the number of sensitive words and the number of characters of sensitive words, the first memory occupancy size of all sensitive words currently stored in the shard service under the Trie data structure and the second memory occupancy size under the dictionary data structure are obtained.

[0075] It should be noted that the query condition is that a full match search is required when searching for sensitive words, that is, the query condition does not require a similarity judgment. Therefore, in order to save memory resources and reduce memory usage, this embodiment obtains the memory usage size under different data structures to achieve flexible conversion and dynamic adjustment of the data structure, which is more flexible and reasonable.

[0076] 4. If the size of the second memory usage is smaller than the size of the first memory usage, converting the data in the Trie data structure into a dictionary data structure.

[0077] Taking Java language (an object-oriented programming language) as an example, a character occupies 3 bytes, and a pointer reference occupies 4 bytes in a 32-bit operating system and 8 bytes in a 64-bit operating system. When the number of sensitive word samples, that is, the number of sensitive words that need to be stored, is very large, other memory usage is negligible. Assume that the number of all sensitive words stored in the current sharding service is 600,000, and the number of different characters of sensitive words is 2 million. Under a 32-bit operating system, if the Trie data structure is used, the memory usage is approximately 2 million (characters) * 4 bytes (references) + 2 million (characters) * 3 bytes (single character) = 14 million bytes, that is, the first memory usage is 14 million bytes. If the dictionary data structure is used, the memory usage is approximately 600,000 (words) * 4 bytes (references) + 2 million (characters) * 3 bytes (single character) = 8.4 million bytes, that is, the second memory usage is 8.4 million bytes. It can be seen that the second memory occupancy size is smaller than the first memory occupancy size. Therefore, without the need for similarity judgment, the data under the Trie data structure of the current shard service is converted into a dictionary data structure to reduce the memory space occupied. This method can better avoid the situation of insufficient memory space. When the memory space is relatively tight, the above-mentioned method of converting the data structure can better realize the release of memory space. The same is true for 64-bit operating systems, which will not be repeated here.

[0078] In some embodiments, the step of obtaining the words to be searched includes:

[0079] 1. Get the text to be searched.

[0080] Second, use the preset filter to filter the text to be searched to obtain the target text. It should be noted that by filtering the text, some meaningless words in the text to be searched can be removed to improve the accuracy of subsequent sensitive word search.

[0081] 3. Using a preset word segmenter, segment the target text to obtain at least one word segment. 4. Perform synonym conversion on the word segment to obtain a target synonym.

[0082] Specifically, using a preset synonym converter, the segmented words are converted into synonyms to obtain target synonyms, which can facilitate subsequent sensitive word searches and help improve the hit rate of sensitive word searches. It is understandable that the segmented words in the text to be searched are not necessarily standard words. By converting the segmented words into synonyms, the accuracy of sensitive word searches can be improved.

[0083] It should be noted that by performing the above-mentioned text filtering, word segmentation and synonym conversion operations, it is possible to help detect or retrieve sensitive words in large texts (large texts). For sensitive word retrieval in large texts, compared with using a database storing sensitive words to perform fuzzy queries on large texts, the above-mentioned method can better improve the efficiency of retrieval for sensitive words in large texts.

[0084] 5. Match the target synonym with the historical sensitive words pre-stored in the cache, where the historical sensitive words are sensitive words that were successfully retrieved previously, and the successful retrieval means that sensitive words that are the same or similar to the previous target synonym are retrieved.

[0085] 6. If the match is successful, the target synonym is determined to be a sensitive word; if the match fails, the target synonym is determined to be the word to be searched.

[0086] It should be noted that if a word to be searched is determined to be a sensitive word, the word to be searched is stored in the cache as a historical sensitive word in the cache. When searching for sensitive words next time, the target synonym is first matched with the historical sensitive words in the cache. If the match is successful, the target synonym is determined to be a sensitive word; if the match fails, it indicates that a subsequent sensitive word search step is required, so the target synonym is determined as the word to be searched. Through the above steps, the search pressure and load pressure of each shard service can be greatly reduced, and the efficiency of sensitive word search can be effectively improved.

[0087] In some embodiments, the step of performing sensitive word retrieval based on the to-be-retrieved word and the data stored in the to-be-retrieved shard service includes:

[0088] 1. Obtain the data structure of the data stored in the shard service to be retrieved.

[0089] 2. If the data structure of the data stored in the to-be-searched shard service is a Trie data structure, the to-be-searched word is split into characters to obtain at least two split characters; based on the split characters and the data stored in the to-be-searched shard service, a similarity judgment is performed to obtain the similarity between the to-be-searched word and the sensitive word stored in the to-be-searched shard service; if the similarity is greater than or equal to a preset similarity threshold, it is determined that the to-be-searched word is a sensitive word, and the to-be-searched word is stored in the cache.

[0090] Specifically, if the data structure of the data stored in the to-be-searched shard service is a Trie data structure, the to-be-searched word is split into characters to obtain at least two split characters. The split characters are sequentially compared with the data stored in the to-be-searched shard service to obtain a matching ratio between the split characters of the to-be-searched word and the sensitive words. The matching ratio is determined as the similarity between the to-be-searched word and the sensitive words stored in the to-be-searched shard service. The similarity threshold can be set according to actual needs, such as 90%.

[0091] 3. If the data structure of the data stored in the to-be-searched shard service is a dictionary data structure, the to-be-searched word is matched with the sensitive words under the dictionary data structure; if the match is successful, the to-be-searched word is determined to be a sensitive word, and the to-be-searched word is stored in the cache.

[0092] It should be mentioned that by storing the to-be-searched words that are determined to be sensitive words in the cache, it is possible to facilitate the subsequent sensitive word search by directly matching the sensitive words in the cache, thereby improving the efficiency of the sensitive word search.

[0093] It is also necessary to mention that if the shard service to be searched includes both data in the Trie data structure and data in the dictionary data structure, the preset query condition is obtained. If the query condition is that a full match search is required when searching for sensitive words, that is, no similarity judgment is required, the sensitive word search is performed based on the data in the dictionary data structure. If the query condition is that a similarity judgment is required when searching for sensitive words, the sensitive word search is performed based on the data in the Trie data structure.

[0094] In some embodiments, it also includes:

[0095] If the first character of the current word to be searched is missing in the target mapping table, the current word to be searched is determined to be a normal word, and the search is stopped or the first character of the next word to be searched is matched with the first character in the target mapping table.

[0096] It should be noted that if the target mapping table lacks the first character of the current search term, it indicates that the current search term is a non-sensitive term, and the search result can be directly returned, and the search can be stopped or the next search term can be searched. The above method can effectively improve the efficiency of sensitive word search and reduce the search pressure and load pressure of each shard service.

[0097] The following is a specific example to further explain the step of storing (adding) sensitive words in the above example.

[0098] Please refer to Figure 2 In order to improve the efficiency of sensitive word storage, this embodiment divides the memory into multiple shard services, such as Figure 2 The first shard service and the second shard service in the shard service cluster. Multiple shard services form a shard service cluster. In addition, in order to further improve the efficiency of sensitive word storage, this embodiment also provides a main service cluster, which includes multiple main services. The multiple main services can receive newly added sensitive words, and perform synonym conversion and mapping relationship (establishing a mapping relationship between the first character and the shard service) on the newly added sensitive words, so as to realize the storage of multiple newly added sensitive words. Multiple main services have a sequential execution order.

[0099] When storing sensitive words (newly added), such as Figure 2 As shown, first, the newly added sensitive words (such as [transfer money], etc.) are input into the main service. Secondly, the synonym converter is used to convert the newly added sensitive words into synonyms, such as the synonym of [transfer money] is [transfer account], etc., to obtain the sensitive words to be stored (here, the synonyms of the newly added sensitive words are used as the sensitive words to be stored. In the actual operation process, the newly added sensitive words themselves can also be used as the sensitive words to be stored, etc.). If this is the first time to store sensitive words, it is necessary to establish a target mapping table, that is, a first character-slice number mapping table. Get the current polling slicing number (if this is the first time to store sensitive words, the current polling slicing number defaults to the slicing number of the first slicing service among multiple slicing services). Based on the polling slicing number, the target slicing service is obtained, that is, the slicing service corresponding to the polling slicing number, such as the first slicing service. Then, a mapping relationship between the first character of the sensitive word to be stored and the slicing number of the target slicing service is established in the target mapping table, such as establishing a mapping relationship between [transfer] and slicing 1. Afterwards, the polling shard number is updated, that is, the polling shard number is updated to the shard number of the next shard service of the current target shard service, so as to be read or obtained when the sensitive word is stored next time. Finally, the sensitive word to be stored [transfer] is stored in the target shard service, that is, the first shard service. Specifically, the target address of the target shard service is obtained; a storage request is sent to the target address, the storage request includes the sensitive word to be stored, and the storage request is used to instruct the target shard service to perform character splitting on the sensitive word to be stored to obtain at least two target characters, such as splitting [transfer] into two target characters [transfer] and [account]; when the preset storage requirement is single data structure storage, the target characters are stored according to the Trie data structure or the dictionary data structure; when the storage requirement is redundant data structure storage, the target characters are stored according to the Trie data structure and the dictionary data structure respectively. Figure 2Only the case where the storage requirement is a single data structure storage is exemplified. Specifically, when the storage requirement is a single data structure storage, the target characters are stored according to the Trie data structure; a preset query condition is obtained; and it is determined whether the query condition requires a similarity judgment. If not, it is determined whether the second memory usage size V2 is smaller than the first memory usage size V1. If the second memory usage size V2 is smaller than the first memory usage size V1, the data under the Trie data structure is converted into a dictionary data structure. Figure 2 Convert [transfer] and [account] under the Trie data structure to [transfer] under the dictionary data structure to reduce memory resource usage.

[0100] Next, add the second new sensitive word (such as Figure 2 [Explosive Rich] is input into the main service. Similarly, [Explosive Rich] is converted into a synonym, and its synonym is still [Explosive Rich]. [Explosive Rich] is used as the current sensitive word to be stored. Secondly, the current value of the polling shard number is obtained to obtain the current target shard service, that is, the shard service corresponding to the current polling shard number, such as shard 2. A mapping relationship between the first character [Explosive Rich] of [Explosive Rich] and shard 2 is established in the target mapping table. In addition, the polling shard number is updated, that is, the polling shard number is updated to the shard number of the next shard service of the current target shard service, so as to be read or obtained when the sensitive word is stored next time. Then [Explosive Rich] is stored in the second shard service. Among them, storing [Explosive Rich] in the second shard service is the same as above, that is, including character splitting, storage according to the Trie data structure, judging whether similarity judgment is required, judging whether the second memory occupancy size V2 is less than the first memory occupancy size V1, and data structure conversion, etc., which will not be repeated here.

[0101] The following is a specific example to further explain the steps of performing sensitive word search in the above embodiment.

[0102] Please refer to Figure 3In the process of searching for sensitive words, first, obtain the text to be searched, such as [Li Ma, transfer money to me], etc. Use the preset filter to filter the text to be searched to obtain the target text, such as filtering [Li Ma, transfer money to me] to obtain [Li Ma transfer money], etc. Use the preset word segmenter to segment the target text to obtain at least one word, such as segmenting [Li Ma transfer money] to obtain two words [Li Ma] and [Transfer money]. Perform synonym conversion on the word segmentation to obtain the target synonym, such as performing synonym conversion on [Li Ma] and [Transfer money] to obtain the target synonyms [Li Ma] and [Transfer money], etc. Secondly, determine whether the target synonym exists in the cache. If it exists, directly return the result, that is, the target synonym is a sensitive word, or the text to be searched contains sensitive words, or [Transfer money] contained in the text to be searched is a sensitive word, etc. If it does not exist, the target synonym is determined as the word to be searched. Determine whether the first character of the word to be searched exists in the target mapping table. If so, obtain the address of the shard service to be searched corresponding to the first character of the word to be searched, such as the address of the first shard service. Finally, send a search instruction to the address of the first shard service to perform sensitive word query and search. Specifically, determine whether the data structure of the data stored in the shard service to be searched (such as the first shard service here) is a Trie data structure. If so, perform character splitting on the word to be searched to obtain at least two split characters, such as splitting [transfer] into [transfer] and [account]. Based on the split characters and the data of the Trie data structure in the shard service to be searched, perform similarity judgment to obtain the similarity between the word to be searched and the sensitive word stored in the shard service to be searched; if the similarity is greater than or equal to the preset similarity threshold, determine that the word to be searched is a sensitive word. If the data structure of the data stored in the shard service to be searched is a dictionary data structure, match the word to be searched with the sensitive word under the dictionary data structure; if the match is successful, determine that the word to be searched is a sensitive word. After executing the above steps, return the result. The main service determines whether there are sensitive words in the text to be searched based on the returned results. If so, the cache is updated, that is, the retrieved sensitive words are updated to the cache as hot words for reading during the next sensitive word search.

[0103] The sensitive word storage and retrieval system provided by the present invention is described below. The sensitive word storage and retrieval system described below and the sensitive word storage and retrieval method described above can be referenced to each other.

[0104] Please refer to Figure 4 The sensitive word storage and retrieval system provided in this embodiment includes:

[0105] The sensitive word storage module 410 is used to store the sensitive words to be stored in the target sharding service through the target mapping table, wherein the target mapping table includes a mapping relationship between the target sharding service and the first character of the sensitive words to be stored, the target sharding service is one of the multiple sharding services in the memory, and the target sharding service is the sharding service currently polled;

[0106] The sensitive word retrieval module 420 is used to determine the to-be-retrieved shard service corresponding to the first character of the to-be-retrieved word based on the target mapping table and the first character of the to-be-retrieved word; and to perform sensitive word retrieval based on the to-be-retrieved word and the data stored in the to-be-retrieved shard service. The sensitive word storage module 410 is connected to the sensitive word retrieval module 420. The sensitive word storage and retrieval system in this embodiment can well improve the storage efficiency of sensitive words, and effectively improve the retrieval efficiency, with low cost and strong feasibility.

[0107] In some embodiments, the sensitive word storage module 410 is also used to update the preset polling shard number to the shard number of the next shard service of the current target shard service after establishing a mapping relationship between the target shard service and the first character of the sensitive word to be stored. The corresponding target shard service for the next sensitive word storage is determined based on the polling shard number.

[0108] In some embodiments, the sensitive word storage module 410 is specifically used to determine the target sharding service based on the first character of the sensitive word to be stored and the target mapping table;

[0109] Obtain the target address of the target shard service;

[0110] Sending a storage request to the target address, the storage request including the sensitive word to be stored, the storage request being used to instruct the target sharding service to perform character splitting on the sensitive word to be stored to obtain at least two target characters;

[0111] When the preset storage requirement is single data structure storage, the target characters are stored according to the Trie data structure or the dictionary data structure; when the storage requirement is redundant data structure storage, the target characters are stored according to the Trie data structure and the dictionary data structure respectively.

[0112] In some embodiments, the sensitive word storage module 410 is further specifically configured to store the target characters according to a Trie data structure or a dictionary data structure when the preset storage requirement is a single data structure storage, including:

[0113] When the preset storage requirement is single data structure storage, the target character is stored according to the Trie data structure;

[0114] Get the preset query conditions;

[0115] In the case where the query condition requires a full match search when performing a sensitive word search, the number of sensitive words and the number of characters of the sensitive words currently stored in the shard service are obtained, wherein the full match search refers to directly matching the words to be searched with the stored sensitive words; based on the number of sensitive words and the number of characters of sensitive words, a first memory occupancy size of all sensitive words currently stored in the shard service under the Trie data structure and a second memory occupancy size under the dictionary data structure are obtained;

[0116] If the second memory occupancy size is smaller than the first memory occupancy size, the data in the Trie data structure is converted into a dictionary data structure.

[0117] In some embodiments, the sensitive word search module 420 is specifically used to obtain the text to be searched;

[0118] Using a preset filter, the text to be searched is filtered to obtain a target text;

[0119] Using a preset word segmenter, segmenting the target text to obtain at least one word segment;

[0120] Performing synonym conversion on the participle to obtain a target synonym;

[0121] Matching the target synonym with historical sensitive words pre-stored in the cache, wherein the historical sensitive words are sensitive words that were successfully retrieved previously, and the successful retrieval means that sensitive words that are the same or similar to the previous target synonym are retrieved;

[0122] If the match is successful, the target synonym is determined to be a sensitive word; if the match fails, the target synonym is determined to be the word to be searched.

[0123] In some embodiments, the sensitive word search module 420 is specifically used to obtain the data structure of the data stored in the shard service to be searched;

[0124] If the data structure of the data stored in the to-be-searched shard service is a Trie data structure, the to-be-searched word is split into characters to obtain at least two split characters; based on the split characters and the data stored in the to-be-searched shard service, a similarity judgment is performed to obtain the similarity between the to-be-searched word and the sensitive word stored in the to-be-searched shard service; if the similarity is greater than or equal to a preset similarity threshold, it is determined that the to-be-searched word is a sensitive word, and the to-be-searched word is stored in a cache;

[0125] If the data structure of the data stored in the to-be-searched shard service is a dictionary data structure, the to-be-searched word is matched with the sensitive words under the dictionary data structure; if the match is successful, the to-be-searched word is determined to be a sensitive word, and the to-be-searched word is stored in the cache.

[0126] In some embodiments, the sensitive word retrieval module 420 is specifically used to determine that the current word to be searched is a normal word if the first character of the current word to be searched is missing in the target mapping table, stop searching or match the first character of the next word to be searched with the first character in the target mapping table.

[0127] In some embodiments, an electronic device is also provided, which may be a server, and its internal structure is shown in FIG. Figure 5 As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of the above method are implemented.

[0128] In some embodiments, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: storing the sensitive words to be stored in a target sharding service through a target mapping table, wherein the target mapping table includes a mapping relationship between the target sharding service and the first character of the sensitive words to be stored, the target sharding service is one of a plurality of sharding services in the memory, and the target sharding service is the sharding service currently polled; based on the target mapping table and the first character of the word to be retrieved, determining the sharding service to be retrieved corresponding to the first character of the word to be retrieved; and performing sensitive word retrieval based on the word to be retrieved and the data stored in the sharding service to be retrieved.

[0129] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: storing the sensitive words to be stored in the target sharding service through a target mapping table, wherein the target mapping table includes a mapping relationship between the target sharding service and the first character of the sensitive words to be stored, the target sharding service is one of multiple sharding services in the memory, and the target sharding service is the sharding service currently polled; based on the target mapping table and the first character of the word to be retrieved, determining the sharding service to be retrieved corresponding to the first character of the word to be retrieved; based on the word to be retrieved and the data stored in the sharding service to be retrieved, performing sensitive word retrieval.

[0130] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or electronic device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible implementation architecture, function and operation of the method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A sensitive word storage and retrieval method, characterized in that: include: The sensitive words to be stored are stored in the target sharding service through the target mapping table, wherein the target mapping table includes a mapping relationship between the target sharding service and the first character of the sensitive words to be stored, the target sharding service is one of the multiple sharding services in the memory, and the target sharding service is the sharding service currently polled; Based on the target mapping table and the first character of the word to be searched, a to-be-searched slice service corresponding to the first character of the word to be searched is determined; based on the word to be searched and the data stored in the to-be-searched slice service, a sensitive word search is performed.

2. The sensitive word storage and retrieval method according to claim 1, characterized in that: The sensitive word to be stored is the newly added sensitive word itself or a synonym of the newly added sensitive word; if the sensitive word is stored for the first time, the target shard service is the first shard service among the multiple shard services; The mapping relationship between the target sharding service and the first character of the sensitive word to be stored is established before the sensitive word to be stored is stored in the target sharding service. After the mapping relationship between the target sharding service and the first character of the sensitive word to be stored is established, the method further includes: The preset polling shard number is updated to the shard number of the next shard service of the current target shard service, and the corresponding target shard service when sensitive words are stored next time is determined based on the polling shard number.

3. The sensitive word storage and retrieval method according to claim 1, characterized in that: The steps of storing the sensitive words to be stored in the target shard service through the target mapping table include: Determining the target sharding service based on the first character of the sensitive word to be stored and the target mapping table; Obtain the target address of the target shard service; Sending a storage request to the target address, the storage request including the sensitive word to be stored, the storage request being used to instruct the target sharding service to perform character splitting on the sensitive word to be stored to obtain at least two target characters; When the preset storage requirement is single data structure storage, the target characters are stored according to the Trie data structure or the dictionary data structure; when the storage requirement is redundant data structure storage, the target characters are stored according to the Trie data structure and the dictionary data structure respectively.

4. The sensitive word storage and retrieval method according to claim 3, characterized in that: When the preset storage requirement is single data structure storage, the step of storing the target characters according to the Trie data structure or the dictionary data structure includes: When the preset storage requirement is single data structure storage, the target character is stored according to the Trie data structure; Get the preset query conditions; In the case where the query condition requires a full match search when performing a sensitive word search, the number of sensitive words and the number of characters of the sensitive words currently stored in the shard service are obtained, wherein the full match search refers to directly matching the words to be searched with the stored sensitive words; based on the number of sensitive words and the number of characters of the sensitive words, a first memory occupancy size of all sensitive words currently stored in the shard service under the Trie data structure and a second memory occupancy size under the dictionary data structure are obtained; If the second memory occupancy size is smaller than the first memory occupancy size, the data in the Trie data structure is converted into a dictionary data structure.

5. The sensitive word storage and retrieval method according to claim 1, characterized in that: The step of obtaining the words to be searched includes: Get the text to be retrieved; Using a preset filter, the text to be searched is filtered to obtain a target text; Using a preset word segmenter, segmenting the target text to obtain at least one word segment; Performing synonym conversion on the participle to obtain a target synonym; Matching the target synonym with historical sensitive words pre-stored in the cache, wherein the historical sensitive words are sensitive words that were successfully retrieved previously, and the successful retrieval means that sensitive words that are the same or similar to the previous target synonym are retrieved; If the match is successful, the target synonym is determined to be a sensitive word; if the match fails, the target synonym is determined to be the word to be searched.

6. The sensitive word storage and retrieval method according to claim 1 or 5, characterized in that: The step of performing sensitive word retrieval based on the to-be-retrieved word and the data stored in the to-be-retrieved slice service includes: Obtaining a data structure of the data stored in the shard service to be retrieved; If the data structure of the data stored in the to-be-searched shard service is a Trie data structure, the to-be-searched word is split into characters to obtain at least two split characters; based on the split characters and the data stored in the to-be-searched shard service, a similarity judgment is performed to obtain the similarity between the to-be-searched word and the sensitive word stored in the to-be-searched shard service; if the similarity is greater than or equal to a preset similarity threshold, it is determined that the to-be-searched word is a sensitive word, and the to-be-searched word is stored in a cache; If the data structure of the data stored in the to-be-searched shard service is a dictionary data structure, the to-be-searched word is matched with the sensitive words under the dictionary data structure; if the match is successful, the to-be-searched word is determined to be a sensitive word, and the to-be-searched word is stored in the cache.

7. The sensitive word storage and retrieval method according to claim 1 or 5, characterized in that: Also includes: If the first character of the current word to be searched is missing in the target mapping table, the current word to be searched is determined to be a normal word, and the search is stopped or the first character of the next word to be searched is matched with the first character in the target mapping table.

8. A sensitive word storage and retrieval system, characterized in that: include: A sensitive word storage module, used to store the sensitive words to be stored in a target sharding service through a target mapping table, wherein the target mapping table includes a mapping relationship between the target sharding service and the first character of the sensitive words to be stored, the target sharding service is one of the multiple sharding services in the memory, and the target sharding service is the sharding service currently polled; The sensitive word search module is used to determine the to-be-searched fragment service corresponding to the first character of the to-be-searched word based on the target mapping table and the first character of the to-be-searched word; and perform sensitive word search based on the to-be-searched word and the data stored in the to-be-searched fragment service.

9. An electronic device, characterized in that: It includes a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute the computer program stored in the memory to implement the sensitive word storage and retrieval method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is used to enable a computer to execute the sensitive word storage and retrieval method as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • System and method for sensitive text recognition

    CN112183087A

  • Targeted sweep method for key-value data storage

    US20210286804A1