Sensitive word filtering method and related device
By constructing a vector knowledge base for sensitive word fragments and using text vectorization processing, the sensitive word fragments and their sensitivity levels are determined, and the problem of poor filtering of sensitive words caused by the lack of semantic understanding in traditional methods is solved, and more accurate and efficient filtering of sensitive words is achieved.
Patent Information
- Application Number
- CN202411681854.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-22
- Publication Date
- 2025-02-28
AI Technical Summary
The traditional sensitive word filtering method lacks semantic understanding ability, resulting in poor filtering of sensitive words, which cannot effectively identify sensitive words in the text or lead to excessive or insufficient filtering.
By constructing a vector knowledge base for sensitive word fragments, the sensitive word fragments and their sensitivity levels are determined using text vectorization processing and vector matching, and then sensitive word filtering is performed.
The effect of improving the filtering of sensitive words through semantic understanding is achieved, and the accurate identification and filtering of sensitive words in the text is improved, and misjudgment and misjudgment in traditional methods are avoided.
Smart Images
Figure CN119537575A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of natural language processing, and in particular to a sensitive word filtering method and related devices. Background Art
[0002] In current natural language processing applications, how to quickly and accurately filter sensitive words is an important research topic. Traditional sensitive word filtering methods usually rely on predefined sensitive word lists and are implemented through simple string matching or keyword retrieval.
[0003] Traditional sensitive word filtering methods have been widely used in early text processing applications, especially on social media, forums, and instant messaging platforms, which can prevent the spread of bad information to a certain extent. However, with the development of natural language processing technology and the increase in the scale of text data, traditional sensitive word filtering methods have poor results due to their lack of semantic understanding.
[0004] Therefore, how to provide a sensitive word filtering method in order to improve the effect of sensitive word filtering by using semantic understanding has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention
[0005] In view of the above problems, the present application provides a sensitive word filtering method and related devices to achieve the purpose of improving the effect of sensitive word filtering by using semantic understanding. The specific scheme is as follows:
[0006] The first aspect of the present application provides a sensitive word filtering method, comprising:
[0007] Obtaining a text to be processed and a pre-built sensitive word segment vector knowledge base; the sensitive word segment vector knowledge base includes vectors corresponding to multiple sensitive word segments, each vector corresponding to a sensitive word segment is used to represent semantic information of the sensitive word segment, and each sensitive word segment includes at least one sensitive word;
[0008] Performing vectorization processing on the text to obtain a vector corresponding to the text, wherein the vector corresponding to the text is used to represent semantic information of the text;
[0009] Determine the sensitive word segment to be processed in the text and the sensitivity level of each sensitive word segment to be processed by at least using the matching degree between the vector corresponding to the text and the vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base;
[0010] Based on the sensitivity level of each of the sensitive word segments to be processed, sensitive word filtering processing is performed on the text to be processed.
[0011] In a possible implementation, the determining of the sensitive word segments to be processed in the text and the sensitivity level of each sensitive word segment to be processed by at least using the matching degree of the vector corresponding to the text and the vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base includes:
[0012] Determine the sensitive word fragment corresponding to the sensitive word fragment vector in the sensitive word fragment vector knowledge base whose matching degree with the vector corresponding to the text meets the preset condition as the sensitive word fragment to be processed in the text;
[0013] Obtaining a matching score corresponding to each of the sensitive word segments to be processed;
[0014] The sensitivity level of each sensitive word segment to be processed is determined by at least using the matching score corresponding to each sensitive word segment to be processed.
[0015] In a possible implementation, determining the sensitivity level of each sensitive word segment to be processed by at least using the matching score corresponding to each sensitive word segment to be processed includes:
[0016] Obtaining a pre-built blacklist of sensitive word segments and a pre-built whitelist of sensitive word segments; the blacklist of sensitive word segments includes a plurality of sensitive word segments that are clearly identified as containing sensitive information; the whitelist of sensitive word segments includes a plurality of sensitive word segments that may be identified as legal expressions;
[0017] The sensitivity level of each sensitive word segment to be processed is determined by using the matching score corresponding to each sensitive word segment to be processed, the pre-constructed sensitive word segment blacklist, and the pre-constructed sensitive word segment whitelist.
[0018] In a possible implementation, determining the sensitivity level of each sensitive word segment to be processed by using the matching score corresponding to each sensitive word segment to be processed, the pre-constructed sensitive word segment blacklist, and the pre-constructed sensitive word segment whitelist includes:
[0019] For each of the sensitive word segments to be processed, determining whether the sensitive word segment to be processed is included in the sensitive word segment blacklist or the sensitive word segment whitelist;
[0020] If the sensitive word to be processed is included in the sensitive word segment blacklist or the sensitive word segment whitelist, determining the sensitivity level of the sensitive word segment to be processed based on the sensitive word segment blacklist or the sensitive word segment whitelist;
[0021] If the sensitive word to be processed is neither included in the sensitive word segment whitelist nor in the sensitive word segment blacklist, the sensitivity level of the sensitive word segment to be processed is determined using the matching score corresponding to the sensitive word segment to be processed.
[0022] In a possible implementation, the sensitive word filtering process is performed on the text to be processed based on the sensitivity level of each sensitive word segment to be processed, including:
[0023] Get the processing strategies corresponding to each sensitivity level;
[0024] Based on the processing strategy corresponding to the sensitivity level of each of the sensitive word segments to be processed, sensitive word filtering processing is performed on the text to be processed.
[0025] In a possible implementation, the sensitive word segment vector knowledge base is constructed by:
[0026] Collect documents related to preset sensitive words;
[0027] Preprocessing the document to obtain a preprocessed document;
[0028] Segmenting the preprocessed document to obtain multiple sensitive word segments;
[0029] For each sensitive word segment, convert the sensitive word segment into a vector to obtain the vector corresponding to the sensitive word segment;
[0030] The vectors corresponding to the various sensitive word fragments are stored in a vector knowledge base, thereby obtaining the sensitive word fragment vector knowledge base.
[0031] In a possible implementation, after performing sensitive word filtering processing on the text to be processed based on the sensitivity level of each sensitive word segment to be processed, the method further includes:
[0032] Obtaining an update instruction, where the update instruction is used to instruct to update a target object, where the target object is any one or more of the sensitive word segment vector knowledge base, the sensitive word segment blacklist, the sensitive word segment whitelist, and processing strategies corresponding to each sensitivity level;
[0033] In response to the update instruction, the target object is updated.
[0034] A second aspect of the present application provides a sensitive word filtering device, comprising:
[0035] An acquisition unit is used to acquire the text to be processed and a pre-built sensitive word segment vector knowledge base; the sensitive word segment vector knowledge base includes vectors corresponding to multiple sensitive word segments, each vector corresponding to a sensitive word segment is used to represent the semantic information of the sensitive word segment, and each sensitive word segment includes at least one sensitive word;
[0036] A vectorization processing unit, used for performing vectorization processing on the text to obtain a vector corresponding to the text, wherein the vector corresponding to the text is used to represent semantic information of the text;
[0037] a determination unit, configured to determine a sensitive word segment to be processed in the text and a sensitivity level of each sensitive word segment to be processed by at least using a matching degree between a vector corresponding to the text and a vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base;
[0038] The text processing unit is used to perform sensitive word filtering processing on the text to be processed based on the sensitivity level of each sensitive word segment to be processed.
[0039] In a possible implementation, the determining unit includes:
[0040] A sensitive word segment to be processed determining unit, used to determine the sensitive word segment corresponding to the sensitive word segment vector in the sensitive word segment vector knowledge base whose matching degree with the vector corresponding to the text meets a preset condition as the sensitive word segment to be processed in the text;
[0041] A matching score acquisition unit, used to acquire a matching score corresponding to each of the sensitive word segments to be processed;
[0042] The sensitivity level determination unit is used to determine the sensitivity level of each sensitive word segment to be processed by at least using the matching score corresponding to each sensitive word segment to be processed.
[0043] In a possible implementation, the sensitivity level determination unit includes:
[0044] A blacklist and whitelist acquisition unit, used to acquire a pre-constructed blacklist of sensitive word segments and a pre-constructed whitelist of sensitive word segments; the blacklist of sensitive word segments includes a plurality of sensitive word segments that are clearly identified as containing sensitive information; the whitelist of sensitive word segments includes a plurality of sensitive word segments that may be identified as legal expressions;
[0045] The sensitivity level determination subunit is used to determine the sensitivity level of each sensitive word segment to be processed by using the matching score corresponding to each sensitive word segment to be processed, the pre-constructed sensitive word segment blacklist and the pre-constructed sensitive word segment whitelist.
[0046] In a possible implementation, the sensitivity level determination subunit is specifically used to:
[0047] For each of the sensitive word segments to be processed, determining whether the sensitive word segment to be processed is included in the sensitive word segment blacklist or the sensitive word segment whitelist;
[0048] If the sensitive word to be processed is included in the sensitive word segment blacklist or the sensitive word segment whitelist, determining the sensitivity level of the sensitive word segment to be processed based on the sensitive word segment blacklist or the sensitive word segment whitelist;
[0049] If the sensitive word to be processed is neither included in the sensitive word segment whitelist nor in the sensitive word segment blacklist, the sensitivity level of the sensitive word segment to be processed is determined using the matching score corresponding to the sensitive word segment to be processed.
[0050] In a possible implementation, the text processing unit is specifically used to:
[0051] Get the processing strategies corresponding to each sensitivity level;
[0052] Based on the processing strategy corresponding to the sensitivity level of each of the sensitive word segments to be processed, sensitive word filtering processing is performed on the text to be processed.
[0053] In a possible implementation, the device further includes: a sensitive word segment vector knowledge base construction unit, wherein the sensitive word segment vector knowledge base construction unit is specifically used to:
[0054] Collect documents related to preset sensitive words;
[0055] Preprocessing the document to obtain a preprocessed document;
[0056] Segmenting the preprocessed document to obtain multiple sensitive word segments;
[0057] For each sensitive word segment, convert the sensitive word segment into a vector to obtain the vector corresponding to the sensitive word segment;
[0058] The vectors corresponding to the various sensitive word fragments are stored in a vector knowledge base, thereby obtaining the sensitive word fragment vector knowledge base.
[0059] In a possible implementation, the apparatus further includes: an updating unit;
[0060] The update unit is used to obtain an update instruction after the text to be processed is filtered for sensitive words based on the sensitivity level of each sensitive word fragment to be processed, and the update instruction is used to instruct to update the target object, and the target object is any one or more of the sensitive word fragment vector knowledge base, the sensitive word fragment blacklist, the sensitive word fragment whitelist and the processing strategies corresponding to each sensitivity level; in response to the update instruction, the target object is updated.
[0061] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the sensitive word filtering method of the first aspect or any implementation of the first aspect.
[0062] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0063] The memory is used to store computer programs;
[0064] The processor is used to execute the computer program so that the electronic device can implement the sensitive word filtering method of the above-mentioned first aspect or any implementation manner of the first aspect.
[0065] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the sensitive word filtering method of the above-mentioned first aspect or any implementation of the first aspect.
[0066] By means of the above technical scheme, the sensitive word filtering method and related devices provided by the present application pre-construct a sensitive word fragment vector knowledge base including vectors corresponding to multiple sensitive word fragments; after obtaining the text to be processed, the text is first vectorized to obtain the vector corresponding to the text. Since the vector corresponding to each sensitive word fragment is used to represent the semantic information of the sensitive word fragment, each sensitive word fragment includes at least one sensitive word; the vector corresponding to the text is used to represent the semantic information of the text, therefore, subsequently at least the matching degree between the vector corresponding to the text and the vector corresponding to each sensitive word fragment in the sensitive word fragment vector knowledge base is used to determine the sensitive word fragment to be processed in the text, as well as the sensitivity level of each sensitive word fragment to be processed; and based on the sensitivity level of each sensitive word fragment to be processed, the text to be processed is subjected to sensitive word filtering processing, which can realize the utilization of semantic understanding, and thus can improve the effect of sensitive word filtering. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.
[0068] Figure 1 A flowchart of a sensitive word filtering method provided in an embodiment of the present application;
[0069] Figure 2 A schematic diagram of the structure of a sensitive word filtering device provided in an embodiment of the present application;
[0070] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0071] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0072] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0073] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0074] In current natural language processing related applications, how to quickly and accurately filter sensitive words is an important research topic.
[0075] Existing sensitive word filtering methods mainly rely on predefined sensitive word lists and implement filtering through string matching or keyword retrieval. These methods usually involve the following steps: first, manually collect and organize a set of sensitive words to form a sensitive word list; then, load the list into the filtering system, and when the text entered by the user needs to be filtered, the system scans the text content line by line and matches it with the entries in the sensitive word list. If a match is found, the sensitive word is processed according to the preset rules, such as deletion, replacement or blocking.
[0076] In the implementation process, regular expressions and keyword blacklist technologies are relied on. Regular expressions can provide complex pattern matching capabilities to identify sensitive words in text. The keyword blacklist is a simple vocabulary that contains all the words that need to be filtered. These technologies can quickly identify sensitive words in the text and process them accordingly. In addition, some systems have also introduced a whitelist mechanism, which allows words in certain special cases to pass the filter even if they are included in the sensitive word list.
[0077] Traditional sensitive word filtering methods have been widely used in early text processing applications, especially on platforms such as social media, forums, and instant messaging. This method can prevent the spread of bad information to a certain extent.
[0078] However, traditional sensitive word filtering methods mainly rely on simple matching of keywords and fail to understand contextual semantics. Therefore, traditional sensitive word filtering methods often lead to over-filtering or under-filtering, resulting in low accuracy of sensitive word filtering and prone to misjudgment (mistaking harmless words for sensitive words) or missed judgment (failure to identify real sensitive words). For example, legitimate expressions containing sensitive words may also be mistakenly blocked, affecting user experience; while some obscure expressions of sensitive words may escape filtering. For example, "apple" may refer to fruit or technology companies in different contexts, and traditional methods cannot distinguish these different semantics.
[0079] In addition, the sensitive word list of the traditional sensitive word filtering method needs to be manually maintained and updated, and once updated, it needs to be reloaded, making it difficult for the system to achieve real-time dynamic adjustment. The system cannot quickly respond and process new sensitive words or information that needs to be temporarily blocked. This method lacks a flexible adjustment mechanism when facing different scenarios and user needs. Moreover, the traditional sensitive word filtering method has low efficiency in retrieval and matching when processing large-scale text data. When it is necessary to process massive user requests, system performance is easily affected, resulting in long response time and even system freezes. This is particularly insufficient in application scenarios that require high real-time and high concurrency.
[0080] In order to solve the above problems, the embodiment of the present application provides a sensitive word filtering method. The sensitive word filtering method of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.
[0081] Reference Figure 1 , Figure 1 A flowchart of a sensitive word filtering method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, a sensitive word filtering method provided in an embodiment of the present application may include steps S101 to S104, and these steps are described in detail below.
[0082] S101: obtaining a text to be processed and a pre-built sensitive word segment vector knowledge base; the sensitive word segment vector knowledge base includes vectors corresponding to multiple sensitive word segments, each vector corresponding to a sensitive word segment is used to represent semantic information of the sensitive word segment, and each sensitive word segment includes at least one sensitive word;
[0083] In the present application, in a possible implementation, the text to be processed may be any text.
[0084] Considering that in the current era of rapid development of artificial intelligence and natural language processing technology, large language models (LLMs) play a vital role in many scenarios such as question-answering systems, intelligent customer service, online education, and information retrieval due to their powerful semantic understanding and generation capabilities. However, the widespread application of these models also brings new challenges, especially for the control of content security and compliance.
[0085] On the one hand, since the large language model is trained based on massive amounts of Internet text, it will inevitably learn some content that contains sensitive information or is not suitable for public discussion. When users ask questions in a question-and-answer scenario, the model may return answers containing sensitive words due to its powerful association and generation capabilities. This may not only lead to a decline in user experience, but may also violate relevant laws and regulations and cause social ethical disputes. On the other hand, with the increasing awareness of network security and personal privacy protection around the world, governments and relevant departments have issued strict laws and regulations, requiring all types of information service providers to take effective measures to ensure the safety and health of platform content. Therefore, real-time monitoring and reasonable constraints on the output of large language models in question-and-answer scenarios have become key issues that need to be solved urgently.
[0086] In another possible implementation, the text to be processed may be relevant text of a large language model question-answering scenario, such as questions input by users and answers generated by a large language model.
[0087] In this application, the sensitive word segment vector knowledge base may be constructed as follows:
[0088] Collect documents related to preset sensitive words; preprocess the documents to obtain preprocessed documents; segment the preprocessed documents to obtain multiple sensitive word fragments; for each sensitive word fragment, convert the sensitive word fragment into a vector to obtain a vector corresponding to the sensitive word fragment; store the vectors corresponding to each sensitive word fragment in a vector knowledge base to obtain the sensitive word fragment vector knowledge base.
[0089] It should be noted that the preset sensitive words may include political, religious, violent, pornographic and other sensitive words. Preprocessing methods include text cleaning, deduplication and format normalization. After preprocessing, the quality and consistency of the document can be guaranteed. For each sensitive word fragment, the word embedding technology can be used to convert the sensitive word fragment into a vector. The vector knowledge base specifically refers to a database that supports vector storage, such as Milvus, ElasticSearch, etc.
[0090] S102: performing vectorization processing on the text to obtain a vector corresponding to the text, where the vector corresponding to the text is used to represent semantic information of the text;
[0091] In the present application, word embedding technology may also be used to vectorize the text to obtain a vector corresponding to the text.
[0092] S103: Determine the sensitive word segment to be processed in the text and the sensitivity level of each sensitive word segment to be processed by at least using the matching degree between the vector corresponding to the text and the vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base;
[0093] In the present application, multiple sensitivity levels may be preset. In a possible implementation, the sensitivity levels may include "low sensitivity" and "high sensitivity".
[0094] S104: Based on the sensitivity level of each of the sensitive word segments to be processed, perform sensitive word filtering processing on the text to be processed.
[0095] In the present application, the processing strategies corresponding to each sensitivity level can be preset. Assuming that the sensitivity levels include "low sensitivity" and "high sensitivity", the processing strategies corresponding to each sensitivity level can be: strict filtering rules (such as blocking or deleting) are implemented for sensitive word fragments with "high sensitivity", and looser processing (such as prompting warnings or light masking) is implemented for sensitive word fragments with "low sensitivity".
[0096] Then, based on the sensitivity level of each sensitive word fragment to be processed, the text to be processed is subjected to sensitive word filtering processing, including: obtaining the processing strategy corresponding to each sensitivity level; based on the processing strategy corresponding to the sensitivity level of each sensitive word fragment to be processed, the text to be processed is subjected to sensitive word filtering processing.
[0097] It should be noted that the text after sensitive word filtering contains no sensitive information.
[0098] The sensitive word filtering method provided in the present embodiment pre-constructs a sensitive word fragment vector knowledge base including vectors corresponding to multiple sensitive word fragments; after obtaining the text to be processed, the text is first vectorized to obtain the vector corresponding to the text. Since the vector corresponding to each sensitive word fragment is used to represent the semantic information of the sensitive word fragment, each sensitive word fragment includes at least one sensitive word; the vector corresponding to the text is used to represent the semantic information of the text, therefore, subsequently at least the degree of match between the vector corresponding to the text and the vector corresponding to each sensitive word fragment in the sensitive word fragment vector knowledge base is used to determine the sensitive word fragment to be processed in the text, as well as the sensitivity level of each sensitive word fragment to be processed; and based on the sensitivity level of each sensitive word fragment to be processed, the text to be processed is subjected to sensitive word filtering processing, which can realize the use of semantic understanding, and thus can improve the effect of sensitive word filtering.
[0099] In another embodiment of the present application, the implementation method of determining the sensitive word fragment to be processed in the text and the sensitivity level of each sensitive word fragment to be processed in step S103 by at least using the matching degree of the vector corresponding to the text and the vector corresponding to each sensitive word fragment in the sensitive word fragment vector knowledge base is described as follows:
[0100] In a possible implementation, at least using the matching degree between the vector corresponding to the text and the vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base, determining the sensitive word segment to be processed in the text and the sensitivity level of each sensitive word segment to be processed includes:
[0101] The sensitive word fragments corresponding to the sensitive word fragment vectors in the sensitive word fragment vector knowledge base, whose matching degree with the vector corresponding to the text meets preset conditions, are determined as the sensitive word fragments to be processed in the text; the matching degree score corresponding to each of the sensitive word fragments to be processed is obtained; and the sensitivity level of each of the sensitive word fragments to be processed is determined by at least using the matching degree score corresponding to each of the sensitive word fragments to be processed.
[0102] In the present application, the similarity (such as cosine similarity) between the vector corresponding to the text and each sensitive word fragment vector in the sensitive word fragment vector knowledge base can be used to determine the matching degree between the vector corresponding to the text and each sensitive word fragment vector in the sensitive word fragment vector knowledge base. The matching degree meeting the preset condition can mean that the similarity is greater than a preset threshold, or the similarity TopN, where the value of N can be set based on the scenario requirements.
[0103] In the present application, the matching score corresponding to the sensitive word segment to be processed may refer to the similarity score corresponding to the sensitive word segment to be processed.
[0104] In the present application, matching score intervals corresponding to different sensitivity levels may be preset.
[0105] In one possible implementation, at least the matching score corresponding to each of the sensitive word fragments to be processed is used to determine the sensitivity level of each of the sensitive word fragments to be processed. It can be that only the matching score corresponding to each of the sensitive word fragments to be processed can be used to determine the corresponding matching score interval, and then determine the sensitivity level of each of the sensitive word fragments to be processed.
[0106] In order to more accurately determine the sensitivity level of the sensitive word fragments to be processed, in the present application, a blacklist of sensitive word fragments and a whitelist of sensitive word fragments may be pre-constructed; the blacklist of sensitive word fragments includes a plurality of sensitive word fragments that are clearly identified as containing sensitive information; the whitelist of sensitive word fragments includes a plurality of sensitive word fragments that may be identified as legal expressions;
[0107] Then, in another possible implementation, determining the sensitivity level of each sensitive word segment to be processed by at least using the matching score corresponding to each sensitive word segment to be processed includes:
[0108] Obtain a pre-constructed blacklist of sensitive word fragments and a pre-constructed whitelist of sensitive word fragments; and determine the sensitivity level of each sensitive word fragment to be processed by using the matching score corresponding to each sensitive word fragment to be processed, the pre-constructed blacklist of sensitive word fragments, and the pre-constructed whitelist of sensitive word fragments.
[0109] Specifically, the using the matching score corresponding to each sensitive word segment to be processed, the pre-built sensitive word segment blacklist, and the pre-built sensitive word segment whitelist to determine the sensitivity level of each sensitive word segment to be processed includes:
[0110] For each of the sensitive word fragments to be processed, determine whether the sensitive word fragment to be processed is included in the sensitive word fragment blacklist or the sensitive word fragment whitelist; if the sensitive word to be processed is included in the sensitive word fragment blacklist or the sensitive word fragment whitelist, determine the sensitivity level of the sensitive word fragment to be processed based on the sensitive word fragment blacklist or the sensitive word fragment whitelist; if the sensitive word to be processed is neither included in the sensitive word fragment whitelist nor in the sensitive word fragment blacklist, determine the sensitivity level of the sensitive word fragment to be processed using the matching score corresponding to the sensitive word fragment to be processed.
[0111] For example, if the sensitive word to be processed is included in the blacklist of sensitive word segments, the sensitivity level of the sensitive word segment to be processed is determined to be "high sensitivity"; if the sensitive word to be processed is included in the whitelist of sensitive word segments, the sensitivity level of the sensitive word segment to be processed is determined to be "low sensitivity";
[0112] In addition, after performing sensitive word filtering processing on the text to be processed based on the sensitivity level of each sensitive word segment to be processed, the method further includes:
[0113] Obtaining an update instruction, where the update instruction is used to instruct to update a target object, where the target object is any one or more of the sensitive word segment vector knowledge base, the sensitive word segment blacklist, the sensitive word segment whitelist, and processing strategies corresponding to each sensitivity level;
[0114] In response to the update instruction, the target object is updated.
[0115] Specifically, a management interface can be set up to allow administrators to perform operations on the update of the target object. After the operation is completed, it will automatically take effect, reducing maintenance costs.
[0116] A sensitive word filtering method provided by an embodiment of the present application is introduced above, and a device for executing the sensitive word filtering method is introduced below.
[0117] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of a sensitive word filtering device provided in an embodiment of the present application. Figure 2 As shown, the sensitive word filtering device includes:
[0118] An acquisition unit 11 is used to acquire a text to be processed and a pre-constructed sensitive word segment vector knowledge base; the sensitive word segment vector knowledge base includes vectors corresponding to multiple sensitive word segments, each vector corresponding to a sensitive word segment is used to represent semantic information of the sensitive word segment, and each sensitive word segment includes at least one sensitive word;
[0119] A vectorization processing unit 12, used for performing vectorization processing on the text to obtain a vector corresponding to the text, wherein the vector corresponding to the text is used to represent semantic information of the text;
[0120] A determination unit 13, configured to determine a sensitive word segment to be processed in the text and a sensitivity level of each sensitive word segment to be processed by at least using a matching degree between a vector corresponding to the text and a vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base;
[0121] The text processing unit 14 is used to perform sensitive word filtering processing on the text to be processed based on the sensitivity level of each sensitive word segment to be processed.
[0122] In a possible implementation, the determining unit includes:
[0123] A sensitive word segment to be processed determining unit, used to determine the sensitive word segment corresponding to the sensitive word segment vector in the sensitive word segment vector knowledge base whose matching degree with the vector corresponding to the text meets a preset condition as the sensitive word segment to be processed in the text;
[0124] A matching score acquisition unit, used to acquire a matching score corresponding to each of the sensitive word segments to be processed;
[0125] The sensitivity level determination unit is used to determine the sensitivity level of each sensitive word segment to be processed by at least using the matching score corresponding to each sensitive word segment to be processed.
[0126] In a possible implementation, the sensitivity level determination unit includes:
[0127] A blacklist and whitelist acquisition unit, used to acquire a pre-constructed blacklist of sensitive word segments and a pre-constructed whitelist of sensitive word segments; the blacklist of sensitive word segments includes a plurality of sensitive word segments that are clearly identified as containing sensitive information; the whitelist of sensitive word segments includes a plurality of sensitive word segments that may be identified as legal expressions;
[0128] The sensitivity level determination subunit is used to determine the sensitivity level of each sensitive word segment to be processed by using the matching score corresponding to each sensitive word segment to be processed, the pre-constructed sensitive word segment blacklist and the pre-constructed sensitive word segment whitelist.
[0129] In a possible implementation, the sensitivity level determination subunit is specifically used to:
[0130] For each of the sensitive word segments to be processed, determining whether the sensitive word segment to be processed is included in the sensitive word segment blacklist or the sensitive word segment whitelist;
[0131] If the sensitive word to be processed is included in the sensitive word segment blacklist or the sensitive word segment whitelist, determining the sensitivity level of the sensitive word segment to be processed based on the sensitive word segment blacklist or the sensitive word segment whitelist;
[0132] If the sensitive word to be processed is neither included in the sensitive word segment whitelist nor in the sensitive word segment blacklist, the sensitivity level of the sensitive word segment to be processed is determined using the matching score corresponding to the sensitive word segment to be processed.
[0133] In a possible implementation, the text processing unit is specifically used to:
[0134] Get the processing strategies corresponding to each sensitivity level;
[0135] Based on the processing strategy corresponding to the sensitivity level of each of the sensitive word segments to be processed, sensitive word filtering processing is performed on the text to be processed.
[0136] In a possible implementation, the device further includes: a sensitive word segment vector knowledge base construction unit, wherein the sensitive word segment vector knowledge base construction unit is specifically used to:
[0137] Collect documents related to preset sensitive words;
[0138] Preprocessing the document to obtain a preprocessed document;
[0139] Segmenting the preprocessed document to obtain multiple sensitive word segments;
[0140] For each sensitive word segment, convert the sensitive word segment into a vector to obtain the vector corresponding to the sensitive word segment;
[0141] The vectors corresponding to the various sensitive word fragments are stored in a vector knowledge base, thereby obtaining the sensitive word fragment vector knowledge base.
[0142] In a possible implementation, the apparatus further includes: an updating unit;
[0143] The update unit is used to obtain an update instruction after the text to be processed is filtered for sensitive words based on the sensitivity level of each sensitive word fragment to be processed, and the update instruction is used to instruct to update the target object, and the target object is any one or more of the sensitive word fragment vector knowledge base, the sensitive word fragment blacklist, the sensitive word fragment whitelist and the processing strategies corresponding to each sensitivity level; in response to the update instruction, the target object is updated.
[0144] The present application also provides an electronic device in an embodiment. Figure 3 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0145] like Figure 3 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0146] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0147] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any sensitive word filtering method provided in the embodiment of the present application.
[0148] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any sensitive word filtering method provided in the embodiment of the present application.
[0149] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0150] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0151] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0152] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. A sensitive word filtering method, characterized in that: include: Obtaining a text to be processed and a pre-built sensitive word segment vector knowledge base; the sensitive word segment vector knowledge base includes vectors corresponding to multiple sensitive word segments, each vector corresponding to a sensitive word segment is used to represent semantic information of the sensitive word segment, and each sensitive word segment includes at least one sensitive word; Performing vectorization processing on the text to obtain a vector corresponding to the text, wherein the vector corresponding to the text is used to represent semantic information of the text; Determine the sensitive word segment to be processed in the text and the sensitivity level of each sensitive word segment to be processed by at least using the matching degree between the vector corresponding to the text and the vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base; Based on the sensitivity level of each of the sensitive word segments to be processed, sensitive word filtering processing is performed on the text to be processed.
2. The method according to claim 1, characterized in that The step of at least using the matching degree between the vector corresponding to the text and the vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base to determine the sensitive word segment to be processed in the text and the sensitivity level of each sensitive word segment to be processed includes: Determine the sensitive word fragment corresponding to the sensitive word fragment vector in the sensitive word fragment vector knowledge base whose matching degree with the vector corresponding to the text meets the preset condition as the sensitive word fragment to be processed in the text; Obtaining a matching score corresponding to each of the sensitive word segments to be processed; The sensitivity level of each sensitive word segment to be processed is determined by at least using the matching score corresponding to each sensitive word segment to be processed.
3. The method according to claim 2, characterized in that The step of determining the sensitivity level of each sensitive word segment to be processed by at least using the matching score corresponding to each sensitive word segment to be processed includes: Obtaining a pre-built blacklist of sensitive word segments and a pre-built whitelist of sensitive word segments; the blacklist of sensitive word segments includes a plurality of sensitive word segments that are clearly identified as containing sensitive information; the whitelist of sensitive word segments includes a plurality of sensitive word segments that may be identified as legal expressions; The sensitivity level of each sensitive word segment to be processed is determined by using the matching score corresponding to each sensitive word segment to be processed, the pre-constructed sensitive word segment blacklist, and the pre-constructed sensitive word segment whitelist.
4. The method according to claim 3, characterized in that The using the matching score corresponding to each of the sensitive word segments to be processed, the pre-built blacklist of sensitive word segments, and the pre-built whitelist of sensitive word segments to determine the sensitivity level of each of the sensitive word segments to be processed includes: For each of the sensitive word segments to be processed, determining whether the sensitive word segment to be processed is included in the sensitive word segment blacklist or the sensitive word segment whitelist; If the sensitive word to be processed is included in the sensitive word segment blacklist or the sensitive word segment whitelist, determining the sensitivity level of the sensitive word segment to be processed based on the sensitive word segment blacklist or the sensitive word segment whitelist; If the sensitive word to be processed is neither included in the sensitive word segment whitelist nor in the sensitive word segment blacklist, the sensitivity level of the sensitive word segment to be processed is determined using the matching score corresponding to the sensitive word segment to be processed.
5. The method according to claim 3, characterized in that: The sensitive word filtering process is performed on the text to be processed based on the sensitivity level of each sensitive word segment to be processed, including: Get the processing strategies corresponding to each sensitivity level; Based on the processing strategy corresponding to the sensitivity level of each of the sensitive word segments to be processed, sensitive word filtering processing is performed on the text to be processed.
6. The method according to claim 1, characterized in that The method for constructing the sensitive word fragment vector knowledge base includes: Collect documents related to preset sensitive words; Preprocessing the document to obtain a preprocessed document; Segmenting the preprocessed document to obtain multiple sensitive word segments; For each sensitive word segment, convert the sensitive word segment into a vector to obtain the vector corresponding to the sensitive word segment; The vectors corresponding to the various sensitive word fragments are stored in a vector knowledge base, thereby obtaining the sensitive word fragment vector knowledge base.
7. The method according to claim 5, characterized in that After performing sensitive word filtering processing on the text to be processed based on the sensitivity level of each sensitive word segment to be processed, the method further includes: Obtaining an update instruction, where the update instruction is used to instruct to update a target object, where the target object is any one or more of the sensitive word segment vector knowledge base, the sensitive word segment blacklist, the sensitive word segment whitelist, and processing strategies corresponding to each sensitivity level; In response to the update instruction, the target object is updated.
8. A sensitive word filtering device, characterized in that: include: An acquisition unit is used to acquire the text to be processed and a pre-built sensitive word segment vector knowledge base; the sensitive word segment vector knowledge base includes vectors corresponding to multiple sensitive word segments, each vector corresponding to a sensitive word segment is used to represent the semantic information of the sensitive word segment, and each sensitive word segment includes at least one sensitive word; A vectorization processing unit, used for performing vectorization processing on the text to obtain a vector corresponding to the text, wherein the vector corresponding to the text is used to represent semantic information of the text; a determination unit, configured to determine a sensitive word segment to be processed in the text and a sensitivity level of each sensitive word segment to be processed by at least using a matching degree between a vector corresponding to the text and a vector corresponding to each sensitive word segment in the sensitive word segment vector knowledge base; The text processing unit is used to perform sensitive word filtering processing on the text to be processed based on the sensitivity level of each sensitive word segment to be processed.
9. A computer program product, characterized in that It includes computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the sensitive word filtering method as described in any one of claims 1 to 7.
10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the sensitive word filtering method as described in any one of claims 1 to 7.
11. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the sensitive word filtering method as described in any one of claims 1 to 7.