Abnormal vocabulary detection method and device, computer equipment and storage medium

Abstract

The invention relates to the field of data processing and financial science and technology, and provides an abnormal vocabulary detection method and device, computer equipment and a storage medium, and the detection method comprises the steps: reading a vocabulary data source, classifying vocabularies in the vocabulary data source according to vocabulary types, and generating a plurality of basic word libraries of different vocabulary types; creating a first word bank and a second word bank for the preset application module; receiving at least one vocabulary category which is selected by the user from the plurality of vocabulary categories and is not allowed to appear in the preset application module, and matching a corresponding basic word bank from the plurality of basic word banks according to the vocabulary category to serve as a target basic word bank; determining a lexicon group corresponding to the preset application module by utilizing the target basic lexicon, the first lexicon and the second lexicon; and receiving a to-be-detected text input by a user, and performing vocabulary detection on the to-be-detected text according to the word bank group corresponding to the preset application module to obtain an abnormal vocabulary list. According to the invention, differentiated abnormal vocabulary detection can be provided for different application modules.

Description

Technical Field

[0001] The present application relates to the fields of data processing and financial technology, and in particular to a method, device, computer equipment and storage medium for detecting abnormal vocabulary. Background Art

[0002] With the progress of the times and the development of science and technology, various websites have emerged and gradually entered the lives of the public. A website refers to a virtual space on the Internet. It is a carrier for displaying, sharing and exchanging information. It consists of a series of web pages, pictures, videos, etc. It can be used to transmit various forms of information and provide various functions and services. For example, financial websites can be used to display financial information and purchase financial products. In order to maintain the order of the website and the security of the content, some sensitive words will be set, and a sensitive word library will be established based on the sensitive words. These sensitive words are usually not allowed to appear on the website to ensure the health and legality of the website content. Therefore, it is necessary to perform sensitive word detection on the text uploaded to the website to avoid the text content containing sensitive words appearing on the website. For example, if a financial website has set the sensitive word "insurance", the word "insurance" is not allowed to appear in the text uploaded to the financial website.

[0003] However, different application modules have different sensitive words because their application scenarios are usually different. The general sensitive word library is difficult to meet the sensitive word detection needs of different application modules. For example, the word "insurance" is a sensitive word in the interactive module of a financial website, but not in the financial management module. Therefore, how to provide differentiated sensitive word detection for different application modules has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method, apparatus, computer device and storage medium for detecting abnormal vocabulary, which can provide differentiated abnormal vocabulary detection for different application modules.

[0005] In a first aspect, the present application provides a method for detecting abnormal vocabulary, the method comprising: reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, generating multiple basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including multiple basic vocabulary libraries of the same vocabulary category; creating a first vocabulary library and a second vocabulary library for a preset application module, the first vocabulary library including multiple first vocabulary libraries that are not allowed to appear in the preset application module, and the second vocabulary library including multiple second vocabulary libraries that are allowed to appear in the preset application module; receiving at least one vocabulary category selected by a user from multiple vocabulary categories that is not allowed to appear in the preset application module, matching the corresponding basic vocabulary library from the multiple basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category; determining the vocabulary library group corresponding to the preset application module using the target basic vocabulary library, the first vocabulary library and the second vocabulary library; receiving a text to be detected input by a user, performing vocabulary detection on the text to be detected according to the vocabulary library group corresponding to the preset application module, and obtaining a list of abnormal vocabulary included in the text to be detected, the abnormal vocabulary list including basic vocabulary and first vocabulary that are not allowed to appear in the preset application module.

[0006] In a second aspect, the present application provides a detection device for abnormal vocabulary, the device comprising: a generation module, used to read a vocabulary data source, classify the vocabulary in the vocabulary data source according to vocabulary categories, and generate multiple basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including multiple basic vocabulary of the same vocabulary category; a creation module, used to create a first vocabulary library and a second vocabulary library for a preset application module, the first vocabulary library including multiple first vocabulary that are not allowed to appear in the preset application module, and the second vocabulary library including multiple second vocabulary that are allowed to appear in the preset application module; a matching module, used to receive at least one vocabulary category selected by a user from multiple vocabulary categories that is not allowed to appear in the preset application module, match the corresponding basic vocabulary library from the multiple basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category, and use the target basic vocabulary library, the first vocabulary library and the second vocabulary library to determine the vocabulary library group corresponding to the preset application module; a detection module, used to receive a text to be detected input by a user, perform vocabulary detection on the text to be detected according to the vocabulary library group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected, the abnormal vocabulary list includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module, and does not include second vocabulary that is allowed to appear in the preset application module.

[0007] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the abnormal vocabulary detection method when executing the computer program.

[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned abnormal vocabulary detection method are implemented.

[0009] The present application creates basic vocabulary libraries of different vocabulary categories, and the basic vocabulary in the basic vocabulary libraries are all vocabulary that are generally set as abnormal words, so that the same basic vocabulary library can be referenced when determining vocabulary library groups for different application modules, effectively reducing the possibility of repeated vocabulary storage, and effectively reducing the storage space and storage cost of the vocabulary library on the server side. Secondly, by creating a first vocabulary library and a second vocabulary library for a preset application module, the first vocabulary in the first vocabulary library is the vocabulary set as an abnormal word in the preset application module, and the second vocabulary in the second vocabulary library is the vocabulary that is often set as an abnormal word but is not set as an abnormal word in the preset application module, using the target basic vocabulary library , the first vocabulary and the second vocabulary determine the vocabulary group corresponding to the preset application module, so that the vocabulary group includes abnormal words of the preset application module, so as to be able to perform abnormal vocabulary detection on the text to be detected in the preset application module, and includes words that may appear in the basic vocabulary but are not abnormal words in the preset application module, so as to improve the accuracy of abnormal vocabulary detection and avoid misjudging abnormal words that are not in this application module, so that the vocabulary group can meet the personalized needs of abnormal vocabulary detection on the text to be detected in the preset application module, and then provide differentiated vocabulary groups for different application modules to meet the detection needs of different application modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0011] Figure 1 is a schematic diagram of an application environment of a method for detecting abnormal words in an embodiment of the present invention;

[0012] Figure 2 is a flow chart of a method for detecting abnormal words in one embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of a structure of an abnormal vocabulary detection device in one embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0015] Figure 5It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] The abnormal vocabulary detection method provided in the embodiment of the present application can be applied to Figure 1In an application environment, the client communicates with the server through a network. The server can read a vocabulary data source, classify the vocabulary in the vocabulary data source according to vocabulary categories, generate multiple basic vocabulary libraries of different vocabulary categories, each basic vocabulary library includes multiple basic vocabulary of the same vocabulary category, and create a first vocabulary library and a second vocabulary library for a preset application module, the first vocabulary library includes multiple first vocabulary that are not allowed to appear in the preset application module, and the second vocabulary library includes multiple second vocabulary that are allowed to appear in the preset application module, and then receive at least one vocabulary category selected by a user from multiple vocabulary categories that is not allowed to appear in the preset application module, match the corresponding basic vocabulary library from the multiple basic vocabulary libraries according to the vocabulary category as a target basic vocabulary library, so as to determine the vocabulary library group corresponding to the preset application module using the target basic vocabulary library, the first vocabulary library and the second vocabulary library, and can also receive a text to be detected input by a user, perform vocabulary detection on the text to be detected according to the vocabulary library group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected, and the abnormal vocabulary list includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module. The present application creates basic vocabulary libraries of different vocabulary categories, and the basic vocabulary in the basic vocabulary libraries are all vocabulary that are generally set as abnormal words, so that the same basic vocabulary library can be referenced when determining vocabulary library groups for different application modules, effectively reducing the possibility of repeated vocabulary storage, and effectively reducing the storage space and storage cost of the vocabulary library on the server side. Secondly, by creating a first vocabulary library and a second vocabulary library for a preset application module, the first vocabulary in the first vocabulary library is the vocabulary set as an abnormal word in the preset application module, and the second vocabulary in the second vocabulary library is the vocabulary that is often set as an abnormal word but is not set as an abnormal word in the preset application module, using the target basic vocabulary library, The first word library and the second word library determine the word library group corresponding to the preset application module, so that the abnormal words of the preset application module are included in the word library group, so that the abnormal vocabulary detection can be performed on the text to be detected of the preset application module, and the vocabulary that may appear in the basic word library, but is not an abnormal word in the preset application module, is included to improve the accuracy of abnormal vocabulary detection, avoid misjudging the vocabulary of abnormal words of non-this application module, so that the word library group can meet the personalized needs of abnormal vocabulary detection of the text to be detected of the preset application module, and then provide differentiated word library groups for different application modules to meet the detection needs of different application modules. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present application is described in detail below through specific embodiments.

[0018] See also Figure 2 As shown, Figure 2A flowchart of a method for detecting abnormal words provided in an embodiment of the present application includes the following steps:

[0019] S100: reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category.

[0020] Sensitive words refer to words that may cause discomfort, disgust or violate regulations in the online environment, such as words with uncivilized or bad tendencies, or words involving false propaganda and other information. In order to maintain the order of the website and the security of the content, and to ensure the health and legality of the website content, it is necessary to manage sensitive words and create a word library. Sensitive words are also called abnormal words.

[0021] The words in the vocabulary data source are words that are generally set as abnormal words, for example, words that are set as abnormal words more often according to statistics of big data technology. The words can be divided into different vocabulary categories according to the reasons why the words are selected as abnormal words. By reading the vocabulary data source and classifying the words in the vocabulary data source according to the vocabulary categories, multiple basic vocabulary libraries are generated, so that the basic words in each basic vocabulary library are all of the same vocabulary category and are all words that are generally set as abnormal words.

[0022] S200: Creating a first word library and a second word library for a preset application module, wherein the first word library includes a plurality of first words that are not allowed to appear in the preset application module, and the second word library includes a plurality of second words that are allowed to appear in the preset application module.

[0023] The multiple first words included in the first vocabulary are not allowed to appear in the preset application module and are abnormal words only for the preset application module. These words are not generally set as abnormal words, but these words are not suitable to appear in the preset application module, so a specific first vocabulary is created for the preset application module.

[0024] For example, words such as "buy", "genuine", and "payment" are frequently used in the online world, but these words are set as abnormal words in the live broadcast module of a certain social networking site, so these words are included in the first vocabulary corresponding to the live broadcast module of the social networking site.

[0025] The multiple second words included in the second vocabulary are words that are allowed to appear in the preset application module, but these words are generally set as abnormal words. In order to avoid including these words in the basic vocabulary, which will result in these allowed words being judged as abnormal words when the basic vocabulary is used to detect abnormal words in the text to be detected, a second vocabulary is created for the preset application module, and these words are called second words and included in the second vocabulary.

[0026] For example, words such as "sales promotion" and "insurance" are often set as abnormal words because they involve commercial promotion, but these words are allowed to appear in the insurance module of a certain financial website, so these words are included in the second vocabulary corresponding to the insurance module of a certain financial website.

[0027] S300: receiving at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in a preset application module, and matching a corresponding basic vocabulary library from a plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category.

[0028] Different preset application modules have different reasons for setting abnormal words due to different functions and application scenarios, so the vocabulary categories that are not allowed to appear in the preset application modules are different, and the basic vocabulary libraries used for abnormal vocabulary detection in the preset application modules are also different. The user selects the vocabulary category that is not allowed to appear in the preset application module from multiple vocabulary categories, and the server receives the vocabulary category selected by the user, and uses the vocabulary category to match the corresponding basic vocabulary library as the target basic vocabulary library, so that the target basic vocabulary library can be used to detect abnormal vocabulary in the text to be detected in the preset application module.

[0029] S400: Determine a vocabulary group corresponding to a preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary.

[0030] The server uses the target basic vocabulary, the first vocabulary and the second vocabulary to determine the vocabulary group corresponding to the preset application module, so that the vocabulary group includes all abnormal words set by the preset application module, as well as words that are allowed to appear in the preset application module but may be generally set as abnormal words by other application modules. The vocabulary group is used to detect abnormal vocabulary in the text to be detected in the preset application module.

[0031] When creating a word library group for a preset application module, the target basic word library is referenced, so that when creating a word library group for a certain application module, only the vocabulary category needs to be selected, and the corresponding abnormal words can be included in the word library group by referencing the basic word library, which effectively improves the efficiency of creating the word library group, and the word library groups of multiple application modules can reference the same basic word library, avoiding repeated storage of the same words when creating word library groups for multiple application modules, effectively reducing the storage space and storage cost of the word library on the server side. By creating a first word library for the preset application module, the abnormal words not included in the target basic word library can be supplemented, avoiding the omission of abnormal words when using the word library group to detect abnormal words in the text to be detected of the preset application module, and creating a second word library for the preset application module can list the words included in the target basic word library, but not abnormal words for the preset application module, so as to exclude these words when using the word library group to detect abnormal words in the text to be detected of the preset application module, which is conducive to improving the accuracy of abnormal word detection.

[0032] In some embodiments, when determining the word library group, the server can determine the word repetition rate of the first word library, so as to remove duplicates from the first word library when the word repetition rate is greater than a preset threshold. For details, see the following steps included in S400.

[0033] S410: Compare the target basic vocabulary with the first vocabulary to obtain a vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary.

[0034] The server compares the target basic vocabulary with the first vocabulary, specifically, compares the basic words in the target basic vocabulary with the first words in the first vocabulary one by one, obtains the repeated words and their numbers between the target basic vocabulary and the first vocabulary, and calculates the vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary, which is equal to the number of repeated words divided by the number of first words.

[0035] In some embodiments, when comparing the target basic vocabulary and the first vocabulary, a lexical semantic comparison method can be used to calculate the semantic similarity between each word in the first vocabulary and each word in the target basic vocabulary. If the semantic similarity between a word in the first vocabulary and a word in the target basic vocabulary is greater than a similarity threshold, it indicates that the two words describe the same content, and the two words are identified as duplicate words.

[0036] S420: Determine whether the vocabulary repetition rate is greater than a preset threshold.

[0037] If so, executing S430: deleting the words repeated in the target basic vocabulary from the first vocabulary to obtain a deduplicated vocabulary.

[0038] The vocabulary repetition rate indicates the ratio of the number of repeated words that appear in both the first vocabulary and the target basic vocabulary to the total number of words in the first vocabulary. If the vocabulary repetition rate is greater than the preset threshold, it indicates that the number of repeated words included in both the first vocabulary and the target basic vocabulary is too large, that is, a large number of words are stored repeatedly, which will lead to a waste of storage resources. Therefore, these words are removed from the first vocabulary to effectively save storage space and reduce storage costs.

[0039] For example, the first vocabulary includes a total of 50 first words, and 15 of these 50 first words are also included in the target basic vocabulary. Therefore, the vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary is 30%, and the preset threshold is 10%. The vocabulary repetition rate is greater than the preset threshold, so these 15 words are removed from the first vocabulary, and the resulting deduplicated vocabulary includes a total of 35 first words.

[0040] S440: forming a vocabulary group corresponding to a preset application module according to the target basic vocabulary, the deduplication vocabulary and the second vocabulary.

[0041] The server uses the target basic vocabulary, the deduplication vocabulary and the second vocabulary to form a vocabulary group corresponding to the preset application module. The target basic vocabulary and the deduplication vocabulary of the vocabulary group include all words that are not allowed to appear in the preset application module, and there are no repeated words between the target basic vocabulary and the deduplication vocabulary, which effectively reduces the storage cost and effectively improves the detection efficiency of abnormal vocabulary detection using the vocabulary group. In addition, the second vocabulary includes words that are allowed to appear in the preset application module, but may be included in the target basic vocabulary because they are generally set as abnormal words, so that these words can be excluded when using the vocabulary group for abnormal vocabulary detection, effectively reducing the possibility of misjudging the words in the text to be detected, which is conducive to improving the reliability and accuracy of vocabulary detection.

[0042] When the vocabulary repetition rate is less than or equal to the preset threshold, executing S450: retaining the vocabulary in the first vocabulary library that is repeated with the target basic vocabulary library.

[0043] S460: forming a vocabulary group corresponding to a preset application module according to the target basic vocabulary, the first vocabulary and the second vocabulary.

[0044] When the vocabulary repetition rate is less than or equal to the preset threshold, it indicates that the number of repeated words included in both the first vocabulary and the target basic vocabulary is not large, and will not cause excessive waste of storage resources. There is no need to deduplicate the first vocabulary, and the vocabulary group corresponding to the preset application module can be directly formed based on the target basic vocabulary, the first vocabulary and the second vocabulary, thereby effectively improving the efficiency of generating vocabulary groups.

[0045] In some embodiments, after the target basic word library is updated, it may be necessary to regenerate the word library group of the preset application module. For details, see the following steps included in S400.

[0046] S470: Determine whether the target basic vocabulary is updated.

[0047] If yes, execute S480: determine whether the updating includes removing some basic words in the target basic vocabulary.

[0048] If so, executing S490: using the updated target basic vocabulary, the first vocabulary, and the second vocabulary to again determine the vocabulary group corresponding to the preset application module.

[0049] After the repeated words in the first vocabulary and the target basic vocabulary are deleted from the first vocabulary, if the target basic vocabulary is updated, it indicates that the basic vocabulary in the target basic vocabulary has undergone certain changes. If some basic vocabulary is removed during the update, it may cause the words originally included in both the target basic vocabulary and the first vocabulary to be first deleted from the first vocabulary, and then removed from the target basic vocabulary due to the update of the target basic vocabulary, resulting in the word not being included in the first vocabulary and the target basic vocabulary, resulting in the word being omitted from the vocabulary group, which may further lead to the omission of abnormal vocabulary detection for the text to be detected. Therefore, it is necessary to generate the vocabulary group again based on the first vocabulary without deleting the repeated words to avoid the omission of words, which is conducive to improving the reliability and accuracy of abnormal vocabulary detection using the vocabulary group.

[0050] For example, the target basic vocabulary and the first vocabulary originally both include the word "insurance". After deduplication of the first vocabulary, the word "insurance" is not included in the deduplication vocabulary. After the target basic vocabulary is updated, the word "insurance" is removed from the target basic vocabulary. At this time, the abnormal vocabulary "insurance" originally belonging to the preset application module is not included in the target basic vocabulary and the deduplication vocabulary. This will result in abnormal vocabulary detection for the text to be detected of the preset application module. If the word "insurance" is in the file to be detected, the server cannot use the vocabulary group to detect the word, resulting in omission of abnormal vocabulary detection. Therefore, after the target basic vocabulary is updated to remove the vocabulary, it is necessary to use the updated target basic vocabulary, the first vocabulary and the second vocabulary to re-determine the vocabulary group corresponding to the preset application module to avoid omission of vocabulary.

[0051] In some embodiments, when using the updated target basic vocabulary, the first vocabulary and the second vocabulary to re-determine the vocabulary group corresponding to the preset application module, it is necessary to compare the updated target basic vocabulary with the first vocabulary to obtain the vocabulary repetition rate of the first vocabulary relative to the updated target basic vocabulary, and determine whether the vocabulary repetition rate is greater than a preset threshold. If so, delete the words repeated with the updated target basic vocabulary from the first vocabulary to obtain an updated deduplicated vocabulary. The vocabulary group corresponding to the preset application module is formed according to the updated target basic vocabulary, the updated deduplicated vocabulary and the second vocabulary. If the vocabulary repetition rate is less than or equal to the preset threshold, the vocabulary group corresponding to the preset application module is directly determined based on the updated target basic vocabulary, the first vocabulary and the second vocabulary.

[0052] S500: Receive a text to be detected input by a user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected, wherein the list of abnormal vocabulary includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module, and does not include second vocabulary that is allowed to appear in the preset application module.

[0053] Different preset application modules correspond to different vocabulary groups. After receiving the text to be detected of the preset application module input by the user, abnormal vocabulary detection can be performed on the text to be detected according to the vocabulary group corresponding to the preset application module to obtain an abnormal vocabulary list. The words in the abnormal vocabulary list are not allowed to appear in the preset application module.

[0054] In some embodiments, the target basic vocabulary and the first vocabulary in the vocabulary group corresponding to the preset application module are used to mark the words in the text to be detected, and then the marked words are matched using the second vocabulary. For details, see the following steps included in S500.

[0055] S510: Mark the words that match the basic words and the first words in the word library group corresponding to the preset application module in the text to be detected, and obtain a marked word list.

[0056] The server performs vocabulary division on the text to be detected to obtain multiple words to be detected included in the text to be detected, uses the basic words in the target basic vocabulary of the vocabulary group and the first words in the first vocabulary to compare with the multiple words to be detected in turn, and marks the words that match the basic words and the first words to obtain a marked vocabulary list, which includes all the basic words and the first words that appear in the text to be detected.

[0057] S520: Deleting the words that match the second words in the word library group corresponding to the preset application module from the marked word list to obtain an abnormal word list.

[0058] The server compares the second vocabulary in the second vocabulary library with the vocabulary in the marked vocabulary list to delete the vocabulary in the marked vocabulary list that matches the second vocabulary, thereby obtaining an abnormal vocabulary list. The abnormal vocabulary list includes all abnormal vocabulary in the text to be detected, and does not include vocabulary that is allowed to appear, thereby accurately and efficiently detecting abnormal vocabulary in the text to be detected.

[0059] In some embodiments, the target basic vocabulary, the first vocabulary, and the second vocabulary in the vocabulary group corresponding to the preset application module are used to detect the vocabulary in the text to be detected, and then the compared list is processed to obtain an abnormal vocabulary list. For details, refer to the following steps included in S500.

[0060] S530: Mark the first vocabulary, the second vocabulary and the basic vocabulary matching the vocabulary group corresponding to the preset application module in the text to be detected, and obtain a first vocabulary list, a second vocabulary list and a basic vocabulary list.

[0061] The server divides the text to be detected into words to obtain multiple words to be detected included in the text to be detected, uses the basic words in the target basic vocabulary, the first words in the first vocabulary and the second words in the second vocabulary to compare with the multiple words to be detected, and marks the words that match the basic words, the first words and the second words, respectively, to obtain the first vocabulary list, the second vocabulary list and the basic vocabulary list.

[0062] S540: Combining the first vocabulary list and the basic vocabulary list to form a marked vocabulary list, deleting the words matching the second words in the second vocabulary list from the marked vocabulary list, and obtaining an abnormal vocabulary list.

[0063] By combining the first vocabulary list and the basic vocabulary list to form a marked vocabulary list, the marked vocabulary list includes all the basic vocabulary and the first vocabulary that appear in the text to be detected, that is, the marked vocabulary list includes all abnormal vocabulary in the text to be detected. Then, the vocabulary in the marked vocabulary list that matches the second vocabulary in the second vocabulary list is deleted, that is, the vocabulary allowed to appear in the preset application module is deleted from the marked vocabulary list, which effectively improves the accuracy of abnormal vocabulary detection.

[0064] By creating basic lexicons of different vocabulary categories, the basic words in the basic lexicon are all words that are generally set as abnormal words, so that the same basic lexicon can be referenced when determining lexicon groups for different application modules, effectively reducing the possibility of repeated storage of words, and effectively reducing the storage space and storage cost of the lexicon on the server side. Secondly, by creating a first lexicon and a second lexicon for a preset application module, the first words in the first lexicon are words that are set as abnormal words in the preset application module, and the second words in the second lexicon are words that are often set as abnormal words but are not set as abnormal words in the preset application module. By using the target basic lexicon, the first lexicon and the second lexicon are used to create a basic lexicon for the preset application module. A word library and a second word library determine a word library group corresponding to a preset application module, so that the word library group includes abnormal words of the preset application module, so as to be able to perform abnormal vocabulary detection on the text to be detected in the preset application module, and includes words that may appear in the basic word library but are not abnormal words in the preset application module, so as to improve the accuracy of abnormal vocabulary detection and avoid misjudging abnormal words that are not in this application module, so that the word library group can meet the personalized needs of abnormal vocabulary detection on the text to be detected in the preset application module, and further provide differentiated word library groups for different application modules to meet the detection needs of different application modules.

[0065] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0066] In one embodiment, a device for detecting abnormal words is provided, and the device for detecting abnormal words corresponds to the method for detecting abnormal words in the above embodiment. Figure 3 As shown, the abnormal vocabulary detection device includes a generation module 101, a creation module 102, a matching module 103, and a detection module 104. The detailed description of each functional module is as follows:

[0067] A generating module 101 is used to read a vocabulary data source, classify the vocabulary in the vocabulary data source according to vocabulary categories, and generate a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category;

[0068] A creation module 102, configured to create a first word library and a second word library for a preset application module, wherein the first word library includes a plurality of first words that are not allowed to appear in the preset application module, and the second word library includes a plurality of second words that are allowed to appear in the preset application module;

[0069] A matching module 103, configured to receive at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in a preset application module, match a corresponding basic vocabulary from a plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category, and determine a vocabulary library group corresponding to the preset application module using the target basic vocabulary library, the first vocabulary library, and the second vocabulary library;

[0070] The detection module 104 is used to receive the text to be detected input by the user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected. The abnormal vocabulary list includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module, and does not include second vocabulary that is allowed to appear in the preset application module.

[0071] In one embodiment, when the matching module 103 uses the target basic vocabulary, the first vocabulary, and the second vocabulary to determine the vocabulary group corresponding to the preset application module, it is used to:

[0072] Comparing the target basic vocabulary with the first vocabulary to obtain a vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary;

[0073] Determine whether the vocabulary repetition rate is greater than a preset threshold;

[0074] If yes, delete the words repeated in the target basic vocabulary from the first vocabulary to obtain a deduplicated vocabulary;

[0075] According to the target basic vocabulary, the deduplication vocabulary and the second vocabulary, a vocabulary group corresponding to the preset application module is formed;

[0076] When the vocabulary repetition rate is less than or equal to a preset threshold, a vocabulary group corresponding to the preset application module is directly formed according to the target basic vocabulary, the first vocabulary and the second vocabulary.

[0077] In one embodiment, when the matching module 103 compares the target basic vocabulary with the first vocabulary to obtain the vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary, it is used to:

[0078] Compare the target basic vocabulary with the first vocabulary to obtain the repeated words and their numbers between the target basic vocabulary and the first vocabulary;

[0079] The vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary is calculated, where the vocabulary repetition rate is equal to the number of repeated words divided by the number of first words.

[0080] In one embodiment, after determining the vocabulary group corresponding to the preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary, the matching module 103 is further configured to:

[0081] Determine whether the target basic vocabulary is updated;

[0082] If so, determining whether the update includes removing some basic words in the target basic vocabulary;

[0083] If so, the updated target basic vocabulary, the first vocabulary and the second vocabulary are used to determine the vocabulary group corresponding to the preset application module again.

[0084] In one embodiment, the detection module 104, when performing vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module to obtain a list of abnormal vocabulary included in the text to be detected, is used to:

[0085] Marking the words matching the basic words and the first words in the word library group corresponding to the preset application module in the text to be detected, and obtaining a marked word list;

[0086] The words matching the second word in the word library group corresponding to the preset application module are deleted from the marked word list to obtain an abnormal word list.

[0087] In one embodiment, the detection module 104, when performing vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module to obtain a list of abnormal vocabulary included in the text to be detected, is used to:

[0088] Marking the first vocabulary, the second vocabulary and the vocabulary matching the basic vocabulary in the vocabulary group corresponding to the preset application module in the text to be detected, respectively, to obtain a first vocabulary list, a second vocabulary list and a basic vocabulary list;

[0089] The first vocabulary list and the basic vocabulary list are combined to form a marked vocabulary list, and words matching the second vocabulary in the second vocabulary list are deleted from the marked vocabulary list to obtain an abnormal vocabulary list.

[0090] The present invention provides a detection device for abnormal vocabulary. By creating basic vocabulary libraries of different vocabulary categories, the basic vocabulary in the basic vocabulary libraries are all vocabulary that are generally set as abnormal words, so that the same basic vocabulary library can be referenced when determining vocabulary library groups for different application modules, effectively reducing the possibility of repeated vocabulary storage, and effectively reducing the storage space and storage cost of the vocabulary library on the server side. Secondly, by creating a first vocabulary library and a second vocabulary library for a preset application module, the first vocabulary in the first vocabulary library is the vocabulary set as an abnormal word in the preset application module, and the second vocabulary in the second vocabulary library is the vocabulary that is often set as an abnormal word but is not set as an abnormal word in the preset application module, The target basic vocabulary, the first vocabulary and the second vocabulary are used to determine the vocabulary group corresponding to the preset application module, so that the vocabulary group includes abnormal words of the preset application module, so as to be able to perform abnormal vocabulary detection on the text to be detected in the preset application module, and includes words that may appear in the basic vocabulary but are not abnormal words in the preset application module, so as to improve the accuracy of abnormal vocabulary detection and avoid misjudging abnormal words that are not in this application module, so that the vocabulary group can meet the personalized needs of abnormal vocabulary detection on the text to be detected in the preset application module, and further provide differentiated vocabulary groups for different application modules to meet the detection needs of different application modules.

[0091] For the specific definition of the abnormal vocabulary detection device, please refer to the definition of the abnormal vocabulary detection method above, which will not be repeated here. Each module in the above abnormal vocabulary detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0092] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a method for detecting abnormal vocabulary are implemented.

[0093] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a method for detecting abnormal vocabulary are implemented.

[0094] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0095] Reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category;

[0096] Creating a first word library and a second word library for a preset application module, wherein the first word library includes a plurality of first words that are not allowed to appear in the preset application module, and the second word library includes a plurality of second words that are allowed to appear in the preset application module;

[0097] receiving at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in a preset application module, and matching a corresponding basic vocabulary from a plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category;

[0098] Determine a vocabulary group corresponding to a preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary;

[0099] Receive the text to be detected input by the user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal words included in the text to be detected, wherein the list of abnormal words includes basic words and first words that are not allowed to appear in the preset application module.

[0100] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0101] Reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category;

[0102] Creating a first word library and a second word library for a preset application module, wherein the first word library includes a plurality of first words that are not allowed to appear in the preset application module, and the second word library includes a plurality of second words that are allowed to appear in the preset application module;

[0103] receiving at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in a preset application module, and matching a corresponding basic vocabulary from a plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category;

[0104] Determine a vocabulary group corresponding to a preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary;

[0105] Receive the text to be detected input by the user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal words included in the text to be detected, wherein the list of abnormal words includes basic words and first words that are not allowed to appear in the preset application module.

[0106] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description of the server side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0107] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0108] Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0109] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0110] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

[0111] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting abnormal words, characterized in that: The detection method comprises: Reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category; Creating a first word library and a second word library for a preset application module, wherein the first word library includes a plurality of first words that are not allowed to appear in the preset application module, and the second word library includes a plurality of second words that are allowed to appear in the preset application module; receiving at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in the preset application module, and matching a corresponding basic vocabulary from the plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category; Determine a vocabulary group corresponding to the preset application module by using the target basic vocabulary, the first vocabulary and the second vocabulary; Receive the text to be detected input by the user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected, wherein the list of abnormal vocabulary includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module.

2. The method for detecting abnormal words according to claim 1, characterized in that: The determining the vocabulary group corresponding to the preset application module by using the target basic vocabulary, the first vocabulary, and the second vocabulary includes: Comparing the target basic vocabulary with the first vocabulary to obtain a vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary; Determining whether the vocabulary repetition rate is greater than a preset threshold; If yes, delete the words repeated in the target basic vocabulary from the first vocabulary to obtain a deduplicated vocabulary; A vocabulary group corresponding to the preset application module is formed according to the target basic vocabulary, the deduplication vocabulary and the second vocabulary.

3. The method for detecting abnormal words according to claim 2, characterized in that: The step of comparing the target basic vocabulary with the first vocabulary to obtain a vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary includes: Comparing the target basic vocabulary with the first vocabulary to obtain the repeated words and their numbers between the target basic vocabulary and the first vocabulary; The vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary is calculated, where the vocabulary repetition rate is equal to the number of the repeated words divided by the number of the first words.

4. The method for detecting abnormal words according to claim 2, wherein: After determining whether the vocabulary repetition rate is greater than a preset threshold, the method further includes: When the vocabulary repetition rate is less than or equal to the preset threshold, retaining the vocabulary in the first vocabulary library that is repeated with the target basic vocabulary library; A vocabulary group corresponding to the preset application module is formed according to the target basic vocabulary, the first vocabulary and the second vocabulary.

5. The method for detecting abnormal words according to claim 1 or 2, characterized in that: The detection method further comprises: Determining whether the target basic vocabulary is updated; If so, determining whether the updating includes removing some basic words in the target basic vocabulary; If so, the updated target basic vocabulary, the first vocabulary and the second vocabulary are used to determine the vocabulary group corresponding to the preset application module again.

6. The method for detecting abnormal words according to claim 1, wherein: The step of performing vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module to obtain a list of abnormal vocabulary included in the text to be detected includes: Marking the words matching the basic words and the first words in the word library group corresponding to the preset application module in the text to be detected to obtain a marked word list; The words matching the second word in the word library group corresponding to the preset application module are deleted from the marked word list to obtain the abnormal word list.

7. The method for detecting abnormal words according to claim 1, wherein: The step of performing vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module to obtain a list of abnormal vocabulary included in the text to be detected includes: Marking the words matching the first words, the second words and the basic words in the word library group corresponding to the preset application module in the text to be detected respectively, to obtain a first word list, a second word list and a basic word list; The first vocabulary list is combined with the basic vocabulary list to form a marked vocabulary list, and words matching the second vocabulary in the second vocabulary list are deleted from the marked vocabulary list to obtain the abnormal vocabulary list.

8. A device for detecting abnormal words, characterized in that: include: A generating module, configured to read a vocabulary data source, classify the vocabulary in the vocabulary data source according to vocabulary categories, and generate a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category; A creation module, configured to create a first word library and a second word library for a preset application module, wherein the first word library includes a plurality of first words that are not allowed to appear in the preset application module, and the second word library includes a plurality of second words that are allowed to appear in the preset application module; a matching module, configured to receive at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in the preset application module, match a corresponding basic vocabulary from the plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category, and determine a vocabulary library group corresponding to the preset application module by using the target basic vocabulary library, the first vocabulary library, and the second vocabulary library; The detection module is used to receive a text to be detected input by a user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected, wherein the list of abnormal vocabulary includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module, and does not include second vocabulary that is allowed to appear in the preset application module.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the abnormal vocabulary detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the abnormal vocabulary detection method according to any one of claims 1 to 7 are implemented.

Images