Sensitive word detection method, device, electronic device and storage medium

Abstract

The present invention relates to the field of artificial intelligence technology, and provides a sensitive word detection method, device, electronic device and storage medium, including: obtaining input text; generating a sensitive word retrieval task for the input text, inputting the sensitive word retrieval task into a pre-trained classification model, and obtaining a retrieval complexity level of the sensitive word retrieval task; searching for candidate sensitive words related to the input text in a sensitive word library according to a retrieval strategy corresponding to the retrieval complexity level; inputting the candidate sensitive words and the input text into a generation model, generating an input text annotated with the candidate sensitive words through the generation model, and performing sensitive word detection on the input text annotated with the candidate sensitive words. The present invention performs sensitive word detection through retrieval enhancement generation technology, which can improve the detection accuracy of sensitive words, and timely shield sensitive words in user input text, thereby ensuring the compliance and security of information dissemination.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a sensitive word detection method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of information technology, the Internet has become an indispensable part of people's daily life and work. Online social platforms, automated content generation systems, customer service robots, and internal corporate communications are increasingly used. While users enjoy the convenience of information exchange, they also face many challenges in the network environment, such as the detection and filtering of sensitive words.

[0003] Traditional sensitive word detection methods usually identify and filter sensitive words based on keyword matching strategies. However, when users use homophones, typos, or creative expressions to describe sensitive words, traditional sensitive word detection methods cannot identify these sensitive words, and the detection accuracy of sensitive words is low. Summary of the invention

[0004] The present invention provides a sensitive word detection method, device, electronic device and storage medium, which are used to solve the defect in the prior art that when users use homophones, typos or creative expressions to describe sensitive words, traditional sensitive word detection methods cannot identify these sensitive words, thereby achieving accurate detection of sensitive words.

[0005] The present invention provides a sensitive word detection method, comprising:

[0006] Get input text;

[0007] Generating a sensitive word retrieval task for the input text, inputting the sensitive word retrieval task into a pre-trained classification model, and obtaining a retrieval complexity level of the sensitive word retrieval task;

[0008] Retrieving candidate sensitive words related to the input text in a sensitive word library according to a retrieval strategy corresponding to the retrieval complexity level;

[0009] The candidate sensitive words and the input text are input into a generation model, the generation model is used to generate an input text annotated with the candidate sensitive words, and sensitive word detection is performed on the input text annotated with the candidate sensitive words to obtain an output text after the detected sensitive words are shielded.

[0010] According to a sensitive word detection method provided by the present invention, searching for candidate sensitive words related to the input text in a sensitive word library according to a search strategy corresponding to the search complexity level includes:

[0011] When the search complexity level of the sensitive word search task is a simple level, according to the keyword search strategy corresponding to the simple level, text matching is performed between the sensitive words in the sensitive word library and the input text to obtain a first candidate sensitive word related to the input text;

[0012] When the retrieval complexity level of the sensitive word retrieval task is a secondary complexity level, according to a sparse retrieval strategy corresponding to the secondary complexity level, the sensitive words in the sensitive word library are associated with the input text through an inverted index to obtain a second candidate sensitive word related to the input text;

[0013] When the retrieval complexity level of the sensitive word retrieval task is a complex level, according to the dense retrieval strategy corresponding to the complex level, the input text is input into a pre-trained deep learning model to obtain a text feature vector, and the sensitive word vector in the sensitive word library is matched with the text feature vector to obtain a third candidate sensitive word related to the input text.

[0014] According to a sensitive word detection method provided by the present invention, the sensitive word detection is performed on the input text marked with the candidate sensitive word, comprising:

[0015] Inserting the sensitive words in the sensitive word library into the dictionary tree, searching for the sensitive words in the input text marked with the candidate sensitive words according to the dictionary tree into which the sensitive words are inserted, so as to obtain the first target sensitive word;

[0016] Converting the sensitive words in the sensitive word library into regular expressions, and searching for the sensitive words in the input text marked with the candidate sensitive words according to the regular expressions to obtain the second target sensitive words;

[0017] Using a pre-trained semantic vector model, converting the words in the input text marked with the candidate sensitive word to obtain a text word vector, converting the sensitive words in the sensitive word library to obtain a sensitive word vector, and matching the text word vector with the sensitive word vector to obtain a third target sensitive word;

[0018] A pre-trained large language model is used to perform semantic analysis on the input text marked with the candidate sensitive word to obtain a semantic analysis result, and words in the semantic analysis result are extracted and matched with the sensitive words in the sensitive word library to obtain a fourth target sensitive word.

[0019] According to a sensitive word detection method provided by the present invention, the sensitive word library is managed in the following manner:

[0020] Get sensitive words after text segmentation;

[0021] Using a pre-trained semantic vector model to encode the sensitive words after the text is segmented, to obtain a sensitive word vector;

[0022] The sensitive word vectors are stored in an open source vector database.

[0023] According to a sensitive word detection method provided by the present invention, before generating a sensitive word retrieval task for the input text, the method further includes:

[0024] Using named entity recognition and regular expressions to identify the time and place information in the input text, and normalizing the time and place information to obtain normalized time and place information;

[0025] Use recommendation algorithms to predict users’ future behaviors based on their historical behavior preferences;

[0026] Based on the normalized time and place information and the user's future behavior, a context understanding result of the input text is obtained, so as to perform sensitive word detection based on the context understanding result of the input text.

[0027] According to a sensitive word detection method provided by the present invention, after obtaining the input text, the method further includes:

[0028] Performing normalization processing on the input text to obtain normalized input text;

[0029] Perform word segmentation on the normalized input text to obtain a word list;

[0030] Filtering stop words in the word list to obtain a filtered word list;

[0031] Performing lemma restoration on the words in the filtered word list to obtain normalized words;

[0032] A preprocessing operation is performed on the input text containing the normalized words to obtain the preprocessed input text, so as to perform sensitive word detection based on the preprocessed input text.

[0033] According to a sensitive word detection method provided by the present invention, the step of normalizing the input text to obtain the normalized input text includes:

[0034] Unifying the character encoding of the input text;

[0035] Eliminate the case differences of the input text after character encoding is unified;

[0036] The special characters in the input text after the difference elimination are removed to obtain the input text after normalization processing.

[0037] The present invention also provides a sensitive word detection device, comprising:

[0038] The acquisition module is used to obtain the input text;

[0039] A classification module, used to generate a sensitive word retrieval task for the input text, input the sensitive word retrieval task into a pre-trained classification model, and obtain a retrieval complexity level of the sensitive word retrieval task;

[0040] A retrieval module, configured to retrieve candidate sensitive words related to the input text in a sensitive word library according to a retrieval strategy corresponding to the retrieval complexity level;

[0041] The detection module is used to input the candidate sensitive words and the input text into a generation model, generate the input text marked with the candidate sensitive words through the generation model, and perform sensitive word detection on the input text marked with the candidate sensitive words to obtain an output text after shielding the detected sensitive words.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned sensitive word detection methods is implemented.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the sensitive word detection method as described in any one of the above is implemented.

[0044] The sensitive word detection method, device, electronic device and storage medium provided by the present invention retrieve candidate sensitive words related to the input text in a sensitive word library according to a retrieval strategy corresponding to the retrieval complexity level of the sensitive word retrieval task for the input text after obtaining the input text, and perform sensitive word detection on the input text marked with the candidate sensitive words through a generation model, thereby improving the detection accuracy of sensitive words through retrieval enhancement generation technology, and timely shielding sensitive words in the user input text, thereby ensuring the compliance and security of information dissemination. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 It is a flowchart of a sensitive word detection method provided by an embodiment of the present invention.

[0047] Figure 2 It is a schematic diagram of various process parts of sensitive word detection provided by an embodiment of the present invention.

[0048] Figure 3 It is a structural schematic diagram of a sensitive word detection device provided by an embodiment of the present invention.

[0049] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Figure 1 is a flow chart of a sensitive word detection method provided by an embodiment of the present invention. Figure 1 The present invention provides a sensitive word detection method, which may include the following steps:

[0052] Step 110, obtaining input text.

[0053] It should be noted that the execution subject of the sensitive word detection method provided in the embodiment of the present invention may be an electronic device, a component in an electronic device, an integrated circuit or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device may be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present invention. The following embodiment of the present invention is described with the server as the execution subject.

[0054] The embodiments of the present invention can be applied to various scenarios such as online social platforms, automatic content generation, and client robots. Specifically, a user can input text, and the server can obtain the text input by the user, detect and identify sensitive words in the text input by the user, and then timely block the sensitive words in the input text, thereby ensuring the compliance and security of information dissemination.

[0055] Step 120: Generate a sensitive word search task for the input text, input the sensitive word search task into a pre-trained classification model, and obtain a search complexity level of the sensitive word search task.

[0056] In the embodiment of the present invention, a prompt project can be constructed, and a large language model can be called to build intelligent routing. Through intelligent routing, the most appropriate search strategy can be dynamically selected according to the characteristics and requirements of the search task.

[0057] Specifically, the search complexity level may include a simple level, a sub-complex level and a complex level. Each search complexity level has a corresponding complexity score threshold range.

[0058] Specifically, after obtaining the input text, the input text and the sensitive word library can be integrated into a task description to construct a sensitive word retrieval task for the input text. The sensitive word retrieval task can be input into a pre-trained classification model to obtain a retrieval complexity score for the sensitive word retrieval task, and the threshold range to which the retrieval complexity score belongs can be determined, thereby obtaining a retrieval complexity level for the sensitive word retrieval task.

[0059] The present invention can judge whether the retrieval task is simple, sub-complex or complex by training a classification model of retrieval complexity. When dealing with simple retrieval tasks, keyword retrieval can be used first; when dealing with more complex retrieval tasks that require in-depth understanding of semantics, sparse retrieval can be switched to; when dealing with complex retrieval tasks that require multi-task allocation, dense retrieval can be used. Through this flexible retrieval strategy, accuracy and coverage can be maximized while ensuring retrieval speed.

[0060] Step 130: searching for candidate sensitive words related to the input text in a sensitive word library according to a search strategy corresponding to the search complexity level.

[0061] Different search complexity levels use different sensitive word search strategies. For example, the search strategy corresponding to the simple level can be a keyword search strategy, the search strategy corresponding to the sub-complexity level can be a sparse search strategy, and the search strategy corresponding to the complex level can be a dense search strategy.

[0062] In an embodiment of the present invention, after determining the retrieval complexity level of the sensitive word retrieval task, a retrieval strategy corresponding to the retrieval complexity level can be adopted to retrieve candidate sensitive words related to the input text in the sensitive word library, thereby quickly locating potential sensitive words related to the text input by the user in the sensitive word library, ensuring the high efficiency and accuracy of the retrieval process.

[0063] It should be noted that the above-mentioned classification of search complexity levels and corresponding search strategies are merely examples of the present invention. Those skilled in the art may classify search complexity levels into different levels and configure different search strategies accordingly according to actual needs, and the present invention does not limit this.

[0064] The embodiment of the present invention can achieve flexible switching of retrieval strategies by adopting different retrieval strategies according to different retrieval complexity levels, which is beneficial to maximizing the accuracy and coverage of sensitive word detection while ensuring the retrieval speed.

[0065] Step 140, input the candidate sensitive words and the input text into a generation model, generate an input text marked with the candidate sensitive words through the generation model, and perform sensitive word detection on the input text marked with the candidate sensitive words to obtain an output text after shielding the detected sensitive words.

[0066] In an embodiment of the present invention, candidate sensitive words and input text can be input into a generation model, and the generation model can closely combine the input text with the sensitive word retrieval results to generate input text marked with candidate sensitive words, so that the generation model can perform an in-depth analysis of the input text marked with candidate sensitive words, thereby mining the implicit sensitive information in the input text, which is conducive to accurately detecting sensitive words in the input text.

[0067] Specifically, the generative model can be used to compare and analyze the input text marked with candidate sensitive words with the sensitive word library in real time to ensure accurate detection and automatic tagging of potential sensitive words, thereby timely shielding sensitive words in the generated text content through real-time monitoring and feedback mechanism.

[0068] For detected sensitive words, shielding processing (such as replacing them with shielding characters), obfuscation processing (such as character replacement), or direct replacement with non-sensitive words can be performed to ensure the compliance and security of the output text content.

[0069] The embodiments of the present invention help to enhance the recognition capability of implicit sensitive information and improve the accuracy of sensitive word detection when performing sensitive word detection on the input text by marking the potentially implicit sensitive word information in the input text.

[0070] After obtaining the input text, the embodiment of the present invention retrieves candidate sensitive words related to the input text in the sensitive word library according to the retrieval strategy corresponding to the retrieval complexity level of the sensitive word retrieval task for the input text, and performs sensitive word detection on the input text marked with the candidate sensitive words through the generation model, thereby using the retrieval enhancement generation technology to perform sensitive word detection, which can improve the detection accuracy of sensitive words and timely shield sensitive words in the user input text, thereby ensuring the compliance and security of information dissemination.

[0071] Based on any of the above embodiments, after obtaining the input text, the following steps may also be included:

[0072] Step S11, normalizing the input text to obtain a normalized input text;

[0073] Step S12, performing word segmentation on the normalized input text to obtain a word list;

[0074] Step S13, filtering the stop words in the word list to obtain a filtered word list;

[0075] Step S14, performing lemma restoration on the words in the filtered word list to obtain normalized words;

[0076] Step S15, performing a preprocessing operation on the input text containing the normalized words to obtain a preprocessed input text, and performing sensitive word detection based on the preprocessed input text.

[0077] Specifically, after obtaining the text input by the user, a series of normalization processes may be performed on the input text to obtain the normalized input text, thereby ensuring the consistency of input texts from different sources.

[0078] Specifically, the Chinese word segmentation tool and the English word segmentation tool can be used to perform word segmentation on the normalized input text to obtain a word list and process compound words and abbreviations.

[0079] Specifically, a pre-prepared stop word list may be used to filter out stop words in the word list that have little impact on the text semantics, so as to reduce noise and obtain a filtered word list.

[0080] Specifically, the words in the filtered word list can be lemmatized, and a lemma restoration tool can be used to restore the words to their basic forms, so as to achieve normalization of different inflected forms and obtain normalized words.

[0081] Specifically, other preprocessing operations may be performed on the input text after word normalization, such as removing redundant spaces, processing numbers, and performing spelling correction, etc., so as to ensure the accuracy of the input text and the effectiveness of subsequent analysis.

[0082] The embodiment of the present invention can reduce noise by performing preprocessing operations on the input text, which is conducive to establishing a foundation for subsequent sensitive word detection, thereby improving the accuracy of sensitive word detection.

[0083] Based on any of the above embodiments, the step of normalizing the input text to obtain the normalized input text may specifically include:

[0084] Step S111, unifying the character encoding of the input text;

[0085] Step S112, eliminating the difference between uppercase and lowercase letters in the input text after the character encoding is unified;

[0086] Step S113, clearing the special characters in the input text after the difference is eliminated, and obtaining the input text after normalization.

[0087] In an embodiment of the present invention, a series of normalization operations on the input text may include a character encoding unification operation, an uppercase and lowercase difference elimination operation, a special character removal operation, and the like.

[0088] Specifically, the character encoding of the input text can be unified, the text can be converted to lowercase or uppercase to eliminate the difference in uppercase and lowercase, and special characters such as punctuation marks and emoticons can be cleared, or special characters can be replaced with standard marks, so as to obtain the input text after normalization.

[0089] The embodiment of the present invention can reduce noise by normalizing the input text, which is conducive to establishing a foundation for subsequent sensitive word detection, thereby improving the accuracy of sensitive word detection.

[0090] Based on any of the above embodiments, the sensitive word library is managed in the following manner: obtaining sensitive words after text segmentation; encoding the sensitive words after text segmentation using a pre-trained semantic vector model to obtain sensitive word vectors; and storing the sensitive word vectors in an open source vector database.

[0091] Milvus database is an open source vector database designed to process large-scale vector data. It can provide powerful vector search capabilities and quickly locate sensitive word entries similar to the query word vector. The semantic vector model (BAAI General Embedding, BGE) is an advanced pre-trained language model that can convert input text into a vector form rich in semantic information.

[0092] In the embodiment of the present invention, after preprocessing the input text, the Milvus database and the BGE model may be introduced to construct and maintain an efficient sensitive word library.

[0093] Specifically, the BGE model can be used to encode the words after text segmentation, generate accurate vector representations for each word, and store these word vectors in the Milvus database. The efficient index structure of the Milvus database can be used to lay the foundation for subsequent similar sensitive word searches and vector retrieval.

[0094] Among them, the sensitive word library can be managed according to strict construction specifications and update processes to ensure the comprehensiveness and timeliness of the sensitive word library. At the same time, it can support dynamic updates of the sensitive word library, and can respond instantly to new sensitive word identification or update requirements of existing sensitive words without restarting or waiting, and quickly adapt to changes in the network language environment. When new text is input, Milvus's vector search function can quickly match the sensitive word vector in the input text to achieve rapid detection of sensitive words. In addition, by continuously collecting user feedback and analyzing search results, the BGE model can be carefully fine-tuned to continuously optimize the quality of word vector generation, thereby improving the accuracy of sensitive word detection.

[0095] The embodiment of the present invention not only improves the management and use efficiency of the sensitive word library, but also enhances the overall efficiency and accuracy of text processing by integrating the pre-trained language model and the vector database.

[0096] Based on any of the above embodiments, before generating the sensitive word search task for the input text, the following steps may also be included:

[0097] Step S21, using named entity recognition and regular expressions to identify the time and place information in the input text, and normalizing the time and place information to obtain normalized time and place information;

[0098] Step S22, using a recommendation algorithm to predict the user's future behavior based on the user's historical behavior preferences;

[0099] Step S23, based on the normalized time and place information and the user's future behavior, obtain the context understanding result of the input text, and perform sensitive word detection based on the context understanding result of the input text.

[0100] In an embodiment of the present invention, after obtaining the text input by the user and before performing sensitive word retrieval, the contextual information of the user input can be deeply analyzed, including time, place, and user behavior patterns, to provide a more accurate sensitive word detection benchmark, which is not only conducive to capturing the implicit meaning and context in the input text, but also conducive to significantly improving the accuracy of sensitive word detection.

[0101] Specifically, when processing time and place information, named entity recognition (NER) technology can be used in combination with a regular expression algorithm to match relevant time and place content in the input text, followed by normalization processing to obtain normalized time and place information.

[0102] In order to more effectively mine potential semantic information, the embodiment of the present invention introduces a recommendation algorithm model to predict the user's future behavior based on the user's historical behavior preferences, thereby further enhancing the depth and breadth of context understanding.

[0103] The embodiments of the present invention can provide a more accurate sensitive word detection benchmark by deeply analyzing context information such as time and place information in the input text and user's future behavior, which is conducive to improving the accuracy of sensitive word detection.

[0104] Based on any of the above embodiments, searching the candidate sensitive words related to the input text in the sensitive word library according to the search strategy corresponding to the search complexity level may specifically include:

[0105] Step 131, when the search complexity level of the sensitive word search task is a simple level, according to the keyword search strategy corresponding to the simple level, text matching is performed between the sensitive words in the sensitive word library and the input text to obtain a first candidate sensitive word related to the input text;

[0106] Step 132, when the retrieval complexity level of the sensitive word retrieval task is a secondary complexity level, according to a sparse retrieval strategy corresponding to the secondary complexity level, the sensitive words in the sensitive word library are associated with the input text through an inverted index to obtain a second candidate sensitive word related to the input text;

[0107] Step 133, when the retrieval complexity level of the sensitive word retrieval task is a complex level, according to the dense retrieval strategy corresponding to the complex level, the input text is input into a pre-trained deep learning model to obtain a text feature vector, and the sensitive word vector in the sensitive word library is matched with the text feature vector to obtain a third candidate sensitive word related to the input text.

[0108] In the embodiment of the present invention, a multi-level search strategy is adopted, and different search strategies are adopted for different search complexity levels, which is beneficial to improving the efficiency and accuracy of the search.

[0109] Specifically, if the retrieval complexity level of the sensitive word retrieval task is simple, a keyword retrieval strategy can be adopted to quickly match the sensitive words in the predefined sensitive word library with the input text to obtain the first candidate sensitive words related to the input text, thereby directly filtering out obviously inappropriate content in the input text.

[0110] Specifically, if the retrieval complexity level of the sensitive word retrieval task is sub-complex, a sparse retrieval strategy can be adopted, using data structures such as inverted indexes to quickly associate sensitive words in the sensitive word library with documents, and obtain second candidate sensitive words related to the input text, thereby achieving rapid retrieval of large-scale data sets, thereby quickly processing large amounts of input text data and improving retrieval speed.

[0111] Specifically, if the retrieval complexity level of the sensitive word retrieval task is complex, a dense retrieval strategy can be adopted. Through the deep learning model, the semantic understanding of the input text content can be performed to identify the inappropriate information implied in the context of the input text, obtain the text feature vector, and quickly match the text feature vector with the sensitive word vector in the sensitive word library to obtain the third candidate sensitive word related to the input text. This can handle more complex contexts and metaphors, enhance the ability to identify potential inappropriate content, and help improve the accuracy and depth of retrieval.

[0112] Based on any of the above embodiments, the performing sensitive word detection on the input text marked with the candidate sensitive word may specifically include:

[0113] Step 141, inserting the sensitive words in the sensitive word library into the dictionary tree, searching for the sensitive words in the input text marked with the candidate sensitive words according to the dictionary tree into which the sensitive words are inserted, so as to obtain the first target sensitive word;

[0114] Step 142, converting the sensitive words in the sensitive word library into regular expressions, searching for sensitive words in the input text marked with the candidate sensitive words according to the regular expressions, to obtain second target sensitive words;

[0115] Step 143: using a pre-trained semantic vector model, converting the words in the input text marked with the candidate sensitive word to obtain a text word vector, converting the sensitive words in the sensitive word library to obtain a sensitive word vector, and matching the text word vector with the sensitive word vector to obtain a third target sensitive word;

[0116] Step 144, using a pre-trained large language model, performs semantic analysis on the input text annotated with the candidate sensitive word to obtain a semantic analysis result, extracts words from the semantic analysis result and matches them with the sensitive words in the sensitive word library to obtain a fourth target sensitive word.

[0117] In an embodiment of the present invention, a model can be generated, using a dictionary tree, regular expressions, BGE models, large language models, etc., to compare the input text marked with candidate sensitive words with a sensitive word library in real time, thereby accurately detecting and automatically marking potential sensitive words in the input text marked with candidate sensitive words.

[0118] Specifically, all sensitive words can be inserted into the dictionary tree, and each character of each sensitive word will be inserted into the dictionary tree in turn, and the end mark will be added at the end of the word. Starting from each character of the input text marked with candidate sensitive words, it can be checked along the path of the dictionary tree whether a complete sensitive word can be found. If a complete word is found, its position is recorded to obtain the first target sensitive word.

[0119] Specifically, all sensitive words can be converted into regular expression patterns to match in the input text marked with candidate sensitive words, and the regular expression library is used to search for matching sensitive words in the input text marked with candidate sensitive words to obtain the second target sensitive word.

[0120] Specifically, the BGE model can be used to convert words in the input text marked with candidate sensitive words into high-dimensional text word vectors, thereby capturing contextual information and semantic information. Each text word vector can be calculated for similarity with the sensitive word vector in the sensitive word library, and potential implicit sensitive words can be detected based on the similarity calculation results to obtain the third target sensitive word.

[0121] Specifically, a prompting project can be designed to guide the large language model to identify sensitive information. Based on the prompting project, the large language model can perform semantic analysis on the input text marked with candidate sensitive words to obtain semantic analysis results, extract and identify implicitly transformed sensitive words from the semantic analysis results, and thus obtain the fourth target sensitive word.

[0122] In some embodiments, the detection accuracy can be continuously optimized through user feedback and new data. Specifically, through an adaptive learning mechanism, the sensitive word detection strategy and algorithm parameters can be automatically adjusted according to the detection results and user feedback.

[0123] In some embodiments, the overall performance of the system can be improved by utilizing technical means such as parallel processing and caching, which helps ensure that the system remains efficient and stable when processing large amounts of data.

[0124] In some embodiments, a user-friendly operation interface may be provided, allowing users to conveniently set system parameters, view detection results, and manage sensitive words.

[0125] The embodiment of the present invention can improve the context relevance and accuracy of sensitive word detection and ensure the real-time performance of detection by applying the retrieval enhancement generation technology to sensitive word detection of input text.

[0126] The embodiment of the present invention supports dynamic updating and immediate effectiveness of sensitive word libraries, as well as an adaptive learning mechanism, which is continuously optimized through user feedback and new data, and is conducive to improving detection accuracy.

[0127] The embodiment of the present invention constructs and manages a sensitive word library, formulates detailed word library construction specifications and update procedures, which not only improves the management and use efficiency of the sensitive word library, but also significantly enhances the overall efficiency and accuracy of text processing.

[0128] The embodiment of the present invention is beneficial to improving the retrieval speed and accuracy through the optimized multi-level retrieval strategy.

[0129] The embodiment of the present invention closely combines the input text with the sensitive word search result through a generation model to generate the input text marked with candidate sensitive words, thereby mining the implicit sensitive information in the input text.

[0130] The embodiments of the present invention can provide effective real-time detection and shielding solutions for sensitive words for various application scenarios, and have broad market application prospects and practical application value. With the continuous advancement and improvement of technology, it can play an important role in maintaining a healthy network environment and promoting the security of information exchange.

[0131] Reference Figure 2 , Figure 2 It is a schematic diagram of various process parts of sensitive word detection provided by an embodiment of the present invention.

[0132] Data preprocessing: Data preprocessing operations can be responsible for converting the original text entered by the user into a format that the system can process. Data preprocessing operations not only include unified character encoding and case conversion, but also include deeper text cleaning, such as removing meaningless spaces and special characters, and word segmentation of text to ensure that each word can be correctly recognized by the system. In addition, data preprocessing operations can also include standardization of numbers and spelling correction to reduce misjudgments caused by input errors.

[0133] Sensitive word search: After the user submits the input text, sensitive word search can be performed quickly. Using efficient algorithms and indexing technology, the input text is quickly scanned to identify and locate potential sensitive words. This process not only relies on the comprehensiveness of the sensitive word library, but also requires the efficiency of the text indexing system to ensure the accuracy and speed of the search.

[0134] Sensitive word detection and blocking: Sensitive word detection and blocking can be responsible for real-time monitoring and comparison of generated text content and entries in the sensitive word library. Once a matching sensitive word is found, it can automatically block or replace the sensitive word according to preset rules or user needs, thereby preventing the spread of sensitive content.

[0135] Contextual understanding: It can provide more accurate sensitive word detection basis by deeply analyzing the contextual information input by users, such as time, location, user behavior habits, etc. Through contextual understanding, it can identify the implicit meaning and context in the text, thereby reducing misjudgments and missed judgments.

[0136] Sensitive word library management: By formulating strict sensitive word library construction specifications and update processes, the comprehensiveness and timeliness of the sensitive word library can be ensured, which is conducive to improving the accuracy of sensitive word detection. At the same time, it supports dynamic updates and immediate effectiveness to cope with the ever-changing social language environment.

[0137] Adaptive learning: By continuously collecting user feedback and new data, the accuracy of sensitive word detection can be continuously optimized. Through the adaptive learning mechanism, the detection strategy and algorithm parameters can be automatically adjusted according to the detection results and user feedback to adapt to the ever-changing detection needs.

[0138] Performance optimization: In order to cope with the challenge of large amounts of data, technical means such as parallel processing and caching can be used to significantly improve the overall performance of the system. Performance optimization can ensure the efficiency and stability of the system when processing large amounts of data.

[0139] User interface: It can provide an intuitive and easy-to-use operation interface, allowing users to easily set system parameters, view detection results and manage sensitive words, thereby ensuring the simplicity and intuitiveness of user operations and improving user experience.

[0140] Feedback and learning: Through a detailed feedback collection mechanism, we record and collect feedback information from every user, including positive comments and improvement suggestions, and use this feedback data to build a closed-loop feedback loop, providing a solid data foundation for continuous optimization and adaptive learning.

[0141] Through the collaborative work of these processes, the embodiments of the present invention can achieve comprehensive monitoring and management of user input content, effectively prevent the spread of sensitive information, and at the same time ensure the efficient operation of the system and the optimization of user experience. With the continuous advancement of technology and the continuous changes in user needs, the embodiments of the present invention can continue to self-learn and optimize to provide more accurate and reliable services.

[0142] The sensitive word detection device provided by the present invention is described below. The sensitive word detection device described below and the sensitive word detection method described above can be referenced to each other.

[0143] Figure 3 is a schematic diagram of the structure of a sensitive word detection device provided by an embodiment of the present invention. Figure 3 The embodiment of the present invention provides a sensitive word detection device, which may specifically include the following modules:

[0144] An acquisition module 310 is used to acquire input text;

[0145] A classification module 320 is used to generate a sensitive word search task for the input text, input the sensitive word search task into a pre-trained classification model, and obtain a search complexity level of the sensitive word search task;

[0146] A retrieval module 330, configured to retrieve candidate sensitive words related to the input text in a sensitive word library according to a retrieval strategy corresponding to the retrieval complexity level;

[0147] The detection module 340 is used to input the candidate sensitive words and the input text into a generation model, generate an input text marked with the candidate sensitive words through the generation model, and perform sensitive word detection on the input text marked with the candidate sensitive words to obtain an output text after shielding the detected sensitive words.

[0148] In an optional embodiment, the retrieval module 330 is specifically used to:

[0149] When the search complexity level of the sensitive word search task is a simple level, according to the keyword search strategy corresponding to the simple level, text matching is performed between the sensitive words in the sensitive word library and the input text to obtain a first candidate sensitive word related to the input text;

[0150] When the retrieval complexity level of the sensitive word retrieval task is a secondary complexity level, according to a sparse retrieval strategy corresponding to the secondary complexity level, the sensitive words in the sensitive word library are associated with the input text through an inverted index to obtain a second candidate sensitive word related to the input text;

[0151] When the retrieval complexity level of the sensitive word retrieval task is a complex level, according to the dense retrieval strategy corresponding to the complex level, the input text is input into a pre-trained deep learning model to obtain a text feature vector, and the sensitive word vector in the sensitive word library is matched with the text feature vector to obtain a third candidate sensitive word related to the input text.

[0152] In an optional embodiment, the detection module 340 is specifically configured to:

[0153] Inserting the sensitive words in the sensitive word library into the dictionary tree, searching for the sensitive words in the input text marked with the candidate sensitive words according to the dictionary tree into which the sensitive words are inserted, so as to obtain the first target sensitive word;

[0154] Converting the sensitive words in the sensitive word library into regular expressions, and searching for the sensitive words in the input text marked with the candidate sensitive words according to the regular expressions to obtain the second target sensitive words;

[0155] Using a pre-trained semantic vector model, converting the words in the input text marked with the candidate sensitive word to obtain a text word vector, converting the sensitive words in the sensitive word library to obtain a sensitive word vector, and matching the text word vector with the sensitive word vector to obtain a third target sensitive word;

[0156] A pre-trained large language model is used to perform semantic analysis on the input text marked with the candidate sensitive word to obtain a semantic analysis result, and words in the semantic analysis result are extracted and matched with the sensitive words in the sensitive word library to obtain a fourth target sensitive word.

[0157] In an optional embodiment, the sensitive word library is managed in the following manner:

[0158] Get sensitive words after text segmentation;

[0159] Using a pre-trained semantic vector model to encode the sensitive words after the text is segmented, to obtain a sensitive word vector;

[0160] The sensitive word vectors are stored in an open source vector database.

[0161] In an optional embodiment, before generating the sensitive word search task for the input text, the device is further used to:

[0162] Using named entity recognition and regular expressions to identify the time and place information in the input text, and normalizing the time and place information to obtain normalized time and place information;

[0163] Use recommendation algorithms to predict users’ future behaviors based on their historical behavior preferences;

[0164] Based on the normalized time and place information and the user's future behavior, a context understanding result of the input text is obtained, so as to perform sensitive word detection based on the context understanding result of the input text.

[0165] In an optional embodiment, after acquiring the input text, the device is further configured to:

[0166] Performing normalization processing on the input text to obtain normalized input text;

[0167] Perform word segmentation on the normalized input text to obtain a word list;

[0168] Filtering stop words in the word list to obtain a filtered word list;

[0169] Performing lemma restoration on the words in the filtered word list to obtain normalized words;

[0170] A preprocessing operation is performed on the input text containing the normalized words to obtain the preprocessed input text, so as to perform sensitive word detection based on the preprocessed input text.

[0171] In an optional embodiment, the device is further used for:

[0172] Unifying the character encoding of the input text;

[0173] Eliminate the case differences of the input text after character encoding is unified;

[0174] The special characters in the input text after the difference elimination are removed to obtain the input text after normalization processing.

[0175] After obtaining the input text, the embodiment of the present invention retrieves candidate sensitive words related to the input text in the sensitive word library according to the retrieval strategy corresponding to the retrieval complexity level of the sensitive word retrieval task for the input text, and performs sensitive word detection on the input text marked with the candidate sensitive words through the generation model, thereby using the retrieval enhancement generation technology to perform sensitive word detection, which can improve the detection accuracy of sensitive words and timely shield sensitive words in the user input text, thereby ensuring the compliance and security of information dissemination.

[0176] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the sensitive word detection method, which includes: obtaining an input text; generating a sensitive word retrieval task for the input text, inputting the sensitive word retrieval task into a pre-trained classification model, and obtaining a retrieval complexity level of the sensitive word retrieval task; according to a retrieval strategy corresponding to the retrieval complexity level, retrieving candidate sensitive words related to the input text in a sensitive word library; inputting the candidate sensitive words and the input text into a generation model, generating an input text annotated with the candidate sensitive words through the generation model, and performing sensitive word detection on the input text annotated with the candidate sensitive words, so as to obtain an output text after shielding the detected sensitive words.

[0177] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0178] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the sensitive word detection method provided by the above-mentioned methods, the method comprising: obtaining an input text; generating a sensitive word retrieval task for the input text, inputting the sensitive word retrieval task into a pre-trained classification model, and obtaining a retrieval complexity level of the sensitive word retrieval task; retrieving candidate sensitive words related to the input text in a sensitive word library according to a retrieval strategy corresponding to the retrieval complexity level; inputting the candidate sensitive words and the input text into a generation model, generating an input text annotated with the candidate sensitive words through the generation model, and performing sensitive word detection on the input text annotated with the candidate sensitive words, so as to obtain an output text after shielding the detected sensitive words.

[0179] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0180] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sensitive word detection method, characterized in that: include: Get input text; Generating a sensitive word retrieval task for the input text, inputting the sensitive word retrieval task into a pre-trained classification model, and obtaining a retrieval complexity level of the sensitive word retrieval task; Retrieving candidate sensitive words related to the input text in a sensitive word library according to a retrieval strategy corresponding to the retrieval complexity level; Inputting the candidate sensitive words and the input text into a generation model, generating an input text annotated with the candidate sensitive words through the generation model, and performing sensitive word detection on the input text annotated with the candidate sensitive words to obtain an output text after shielding the detected sensitive words; The step of searching the sensitive word library for candidate sensitive words related to the input text according to the search strategy corresponding to the search complexity level includes: When the search complexity level of the sensitive word search task is a simple level, according to the keyword search strategy corresponding to the simple level, text matching is performed between the sensitive words in the sensitive word library and the input text to obtain a first candidate sensitive word related to the input text; When the retrieval complexity level of the sensitive word retrieval task is a secondary complexity level, according to a sparse retrieval strategy corresponding to the secondary complexity level, the sensitive words in the sensitive word library are associated with the input text through an inverted index to obtain a second candidate sensitive word related to the input text; When the retrieval complexity level of the sensitive word retrieval task is a complex level, according to the dense retrieval strategy corresponding to the complex level, the input text is input into a pre-trained deep learning model to obtain a text feature vector, and the sensitive word vector in the sensitive word library is matched with the text feature vector to obtain a third candidate sensitive word related to the input text.

2. The sensitive word detection method according to claim 1, characterized in that: The performing sensitive word detection on the input text marked with the candidate sensitive words includes: Inserting the sensitive words in the sensitive word library into the dictionary tree, searching for the sensitive words in the input text marked with the candidate sensitive words according to the dictionary tree into which the sensitive words are inserted, so as to obtain the first target sensitive word; Converting the sensitive words in the sensitive word library into regular expressions, and searching for the sensitive words in the input text marked with the candidate sensitive words according to the regular expressions to obtain the second target sensitive words; Using a pre-trained semantic vector model, converting the words in the input text marked with the candidate sensitive word to obtain a text word vector, converting the sensitive words in the sensitive word library to obtain a sensitive word vector, and matching the text word vector with the sensitive word vector to obtain a third target sensitive word; A pre-trained large language model is used to perform semantic analysis on the input text marked with the candidate sensitive word to obtain a semantic analysis result, and words in the semantic analysis result are extracted and matched with the sensitive words in the sensitive word library to obtain a fourth target sensitive word.

3. The sensitive word detection method according to claim 1, characterized in that: The sensitive word library is managed in the following ways: Get sensitive words after text segmentation; Using a pre-trained semantic vector model to encode the sensitive words after the text is segmented, to obtain a sensitive word vector; The sensitive word vectors are stored in an open source vector database.

4. The sensitive word detection method according to claim 1, characterized in that: Before generating the sensitive word search task for the input text, the method further includes: Using named entity recognition and regular expressions to identify the time and place information in the input text, and normalizing the time and place information to obtain normalized time and place information; Use recommendation algorithms to predict users’ future behaviors based on their historical behavior preferences; Based on the normalized time and place information and the user's future behavior, a context understanding result of the input text is obtained, so as to perform sensitive word detection based on the context understanding result of the input text.

5. The sensitive word detection method according to claim 1, characterized in that: After obtaining the input text, the method further includes: Performing normalization processing on the input text to obtain normalized input text; Perform word segmentation on the normalized input text to obtain a word list; Filtering stop words in the word list to obtain a filtered word list; Performing lemma restoration on the words in the filtered word list to obtain normalized words; A preprocessing operation is performed on the input text containing the normalized words to obtain the preprocessed input text, so as to perform sensitive word detection based on the preprocessed input text.

6. The sensitive word detection method according to claim 5, characterized in that: The step of normalizing the input text to obtain the normalized input text includes: Unifying the character encoding of the input text; Eliminate the case differences of the input text after character encoding is unified; The special characters in the input text after the difference elimination are removed to obtain the input text after normalization processing.

7. A sensitive word detection device, characterized in that: include: The acquisition module is used to obtain the input text; A classification module, used to generate a sensitive word retrieval task for the input text, input the sensitive word retrieval task into a pre-trained classification model, and obtain a retrieval complexity level of the sensitive word retrieval task; A retrieval module, configured to retrieve candidate sensitive words related to the input text in a sensitive word library according to a retrieval strategy corresponding to the retrieval complexity level; A detection module, used for inputting the candidate sensitive words and the input text into a generation model, generating an input text annotated with the candidate sensitive words through the generation model, and performing sensitive word detection on the input text annotated with the candidate sensitive words to obtain an output text after shielding the detected sensitive words; Wherein, the retrieval module is specifically used for: When the search complexity level of the sensitive word search task is a simple level, according to the keyword search strategy corresponding to the simple level, text matching is performed between the sensitive words in the sensitive word library and the input text to obtain a first candidate sensitive word related to the input text; When the retrieval complexity level of the sensitive word retrieval task is a secondary complexity level, according to a sparse retrieval strategy corresponding to the secondary complexity level, the sensitive words in the sensitive word library are associated with the input text through an inverted index to obtain a second candidate sensitive word related to the input text; When the retrieval complexity level of the sensitive word retrieval task is a complex level, according to the dense retrieval strategy corresponding to the complex level, the input text is input into a pre-trained deep learning model to obtain a text feature vector, and the sensitive word vector in the sensitive word library is matched with the text feature vector to obtain a third candidate sensitive word related to the input text.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the sensitive word detection method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sensitive word detection method as described in any one of claims 1 to 6 is implemented.

Images