Abnormal vocabulary detection method, device, computer equipment and storage medium

Abstract

The present application relates to the fields of data processing and financial technology, and provides a method, apparatus, computer equipment, and storage medium for detecting abnormal vocabulary. The detection method includes: reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating multiple basic vocabulary libraries of different vocabulary categories; creating a first vocabulary library and a second vocabulary library for a preset application module; receiving at least one vocabulary category selected by a user from multiple vocabulary categories that is not allowed to appear in the preset application module, matching the corresponding basic vocabulary library from multiple basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category; determining the vocabulary library group corresponding to the preset application module using the target basic vocabulary library, the first vocabulary library, and the second vocabulary library; receiving a text to be detected input by the user, performing vocabulary detection on the text to be detected according to the vocabulary library group corresponding to the preset application module, and obtaining a list of abnormal vocabulary. The present application can provide differentiated abnormal vocabulary detection for different application modules.

Description

Technical Field

[0001] The present application relates to the fields of data processing and financial technology, and in particular to a method, apparatus, computer equipment and storage medium for detecting abnormal vocabulary. Background Art

[0002] With the advancement of technology and the progress of the times, various websites have emerged and gradually become part of people's lives. A website is a virtual space on the internet used to display, share, and exchange information. It consists of a series of web pages, images, videos, and other content, and can be used to convey various forms of information and provide various functions and services. For example, financial websites can be used to display financial information and purchase financial products. To maintain the order and security of website content, sensitive words are set and a sensitive word library is established based on these sensitive words. These sensitive words are generally not allowed to appear on websites to ensure the health and legality of website content. Therefore, text uploaded to the website is required to be checked for sensitive words to prevent content containing sensitive words from appearing on the website. For example, if a financial website has the sensitive word "insurance" set, the word "insurance" will not be allowed to appear in text uploaded to the financial website.

[0003] However, different application modules often have different sensitive word settings due to their diverse application scenarios. A universal sensitive word library cannot meet the sensitive word detection needs of these different application modules. For example, the word "insurance" is a sensitive word in the interactive module of a financial website, but not in the wealth management module. Therefore, how to provide differentiated sensitive word detection for different application modules has become a pressing technical challenge. Summary of the Invention

[0004] The present application provides a method, apparatus, computer device, and storage medium for detecting abnormal vocabulary, which can provide differentiated abnormal vocabulary detection for different application modules.

[0005] In a first aspect, the present application provides a method for detecting abnormal vocabulary, the method comprising: reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, generating multiple basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including multiple basic vocabulary libraries of the same vocabulary category; creating a first vocabulary library and a second vocabulary library for a preset application module, the first vocabulary library including multiple first vocabulary libraries that are not allowed to appear in the preset application module, and the second vocabulary library including multiple second vocabulary libraries that are allowed to appear in the preset application module; receiving at least one vocabulary category selected by a user from multiple vocabulary categories that is not allowed to appear in the preset application module, and matching the corresponding basic vocabulary library from the multiple basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category; using the target basic vocabulary library, the first vocabulary library and the second vocabulary library to determine the vocabulary library group corresponding to the preset application module; receiving a text to be detected input by the user, performing vocabulary detection on the text to be detected according to the vocabulary library group corresponding to the preset application module, and obtaining a list of abnormal vocabulary included in the text to be detected, the abnormal vocabulary list including basic vocabulary and first vocabulary that are not allowed to appear in the preset application module.

[0006] In a second aspect, the present application provides a device for detecting abnormal vocabulary, the device comprising: a generation module for reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating multiple basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including multiple basic vocabulary libraries of the same vocabulary category; a creation module for creating a first vocabulary library and a second vocabulary library for a preset application module, the first vocabulary library including multiple first vocabulary libraries that are not allowed to appear in the preset application module, and the second vocabulary library including multiple second vocabulary libraries that are allowed to appear in the preset application module; a matching module for receiving at least one vocabulary category selected by a user from multiple vocabulary categories that is not allowed to appear in the preset application module, matching the corresponding basic vocabulary library from the multiple basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category, and determining the vocabulary library group corresponding to the preset application module using the target basic vocabulary library, the first vocabulary library, and the second vocabulary library; a detection module for receiving a text to be detected input by a user, performing vocabulary detection on the text to be detected according to the vocabulary library group corresponding to the preset application module, and obtaining a list of abnormal vocabulary included in the text to be detected, the abnormal vocabulary list including the basic vocabulary and first vocabulary libraries that are not allowed to appear in the preset application module, and excluding the second vocabulary libraries that are allowed to appear in the preset application module.

[0007] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned abnormal vocabulary detection method when executing the computer program.

[0008] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned abnormal vocabulary detection method are implemented.

[0009] The present application creates basic vocabulary libraries of different vocabulary categories. The basic vocabulary in the basic vocabulary libraries are all vocabulary that are generally set as abnormal words, so that the same basic vocabulary library can be referenced when determining vocabulary library groups for different application modules, effectively reducing the possibility of repeated vocabulary storage and effectively reducing the storage space and storage cost of the vocabulary library on the server side. Secondly, by creating a first vocabulary library and a second vocabulary library for the preset application module, the first vocabulary in the first vocabulary library is the vocabulary set as abnormal word in the preset application module, and the second vocabulary in the second vocabulary library is the vocabulary that is often set as abnormal word but is not set as abnormal word in the preset application module, using the target basic vocabulary library , the first vocabulary and the second vocabulary determine the vocabulary group corresponding to the preset application module, so that the vocabulary group includes abnormal words of the preset application module, so that abnormal vocabulary detection can be performed on the text to be detected in the preset application module, and includes words that may appear in the basic vocabulary but are not abnormal words in the preset application module, so as to improve the accuracy of abnormal vocabulary detection and avoid misjudging abnormal words that are not in this application module, so that the vocabulary group can meet the personalized needs of abnormal vocabulary detection on the text to be detected in the preset application module, and then provide differentiated vocabulary groups for different application modules to meet the detection needs of different application modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 1 is a schematic diagram of an application environment of a method for detecting abnormal vocabulary according to an embodiment of the present invention;

[0012] Figure 2 1 is a flow chart of a method for detecting abnormal words in one embodiment of the present invention;

[0013] Figure 3 1 is a schematic structural diagram of an abnormal vocabulary detection device according to an embodiment of the present invention;

[0014] Figure 4 is a structural diagram of a computer device in one embodiment of the present invention;

[0015] Figure 5FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] The abnormal vocabulary detection method provided in the embodiment of the present application can be applied to Figure 1In an application environment, the client communicates with the server via a network. The server can read a vocabulary data source, classify the vocabulary in the vocabulary data source according to vocabulary categories, generate multiple basic vocabulary libraries of different vocabulary categories, each basic vocabulary library includes multiple basic vocabulary of the same vocabulary category, and create a first vocabulary library and a second vocabulary library for a preset application module, wherein the first vocabulary library includes multiple first vocabulary that are not allowed to appear in the preset application module, and the second vocabulary library includes multiple second vocabulary that are allowed to appear in the preset application module. Then, at least one vocabulary category that is not allowed to appear in the preset application module is selected by the user from multiple vocabulary categories, and a corresponding basic vocabulary library is matched from the multiple basic vocabulary libraries according to the vocabulary category as a target basic vocabulary library, so as to determine the vocabulary library group corresponding to the preset application module using the target basic vocabulary library, the first vocabulary library, and the second vocabulary library. The server can also receive a text to be detected input by the user, perform vocabulary detection on the text to be detected according to the vocabulary library group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected, wherein the abnormal vocabulary list includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module. The present application creates basic vocabulary libraries of different vocabulary categories. The basic vocabulary in the basic vocabulary libraries are all vocabulary that are generally set as abnormal words, so that the same basic vocabulary library can be referenced when determining vocabulary library groups for different application modules, effectively reducing the possibility of repeated vocabulary storage and effectively reducing the storage space and storage cost of the vocabulary library on the server side. Secondly, by creating a first vocabulary library and a second vocabulary library for the preset application module, the first vocabulary in the first vocabulary library is the vocabulary set as an abnormal word in the preset application module, and the second vocabulary in the second vocabulary library is the vocabulary that is often set as an abnormal word but is not set as an abnormal word in the preset application module, using the target basic vocabulary library, The first vocabulary and the second vocabulary determine the vocabulary group corresponding to the preset application module, so that the abnormal words of the preset application module are included in the vocabulary group, so that the text to be detected of the preset application module can be detected with abnormal vocabulary, and the vocabulary that may appear in the basic vocabulary but is not an abnormal word in the preset application module is included, so as to improve the accuracy of abnormal vocabulary detection and avoid misjudging the vocabulary of abnormal words that are not in this application module, so that the vocabulary group can meet the personalized needs of abnormal vocabulary detection of the text to be detected of the preset application module, and then provide differentiated vocabulary groups for different application modules to meet the detection needs of different application modules. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present application is described in detail below through specific embodiments.

[0018] See also Figure 2 As shown, Figure 2A flowchart of a method for detecting abnormal vocabulary provided in an embodiment of the present application includes the following steps:

[0019] S100: Reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category.

[0020] Sensitive words refer to words that may cause discomfort, disgust, or violate regulations online. These words include uncivilized or negative terms, or terms that involve false advertising. To maintain website order and content security, and to ensure the health and legality of website content, it is necessary to manage sensitive words and create a vocabulary. Sensitive words are also called abnormal words.

[0021] The words in the vocabulary data source are generally designated as abnormal words. For example, they may be words that have been designated as abnormal words frequently using big data technology. The words may be divided into different vocabulary categories based on the reasons for their selection as abnormal words. By reading the vocabulary data source and classifying the words in the vocabulary data source according to vocabulary categories, multiple basic vocabulary libraries are generated. Therefore, the basic words in each basic vocabulary library are all of the same vocabulary category and are generally designated as abnormal words.

[0022] S200: Creating a first vocabulary and a second vocabulary for a preset application module, wherein the first vocabulary includes a plurality of first words that are not allowed to appear in the preset application module, and the second vocabulary includes a plurality of second words that are allowed to appear in the preset application module.

[0023] Multiple first words included in the first vocabulary are not allowed to appear in the preset application module and are abnormal words only for the preset application module. These words are not generally set as abnormal words, but these words are not suitable to appear in the preset application module. Therefore, a specific first vocabulary is created for the preset application module.

[0024] For example, words such as "buy", "genuine", and "payment" are frequently used in the online world, but these words will be set as abnormal words in the live broadcast module of a certain social networking site, so these words are included in the first vocabulary corresponding to the live broadcast module of the social networking site.

[0025] The multiple second words included in the second vocabulary are words that are allowed to appear in the preset application module, but these words are generally set as abnormal words. In order to avoid including these words in the basic vocabulary, which will lead to these allowed words being judged as abnormal words when the basic vocabulary is used to detect abnormal words in the text to be detected, a second vocabulary is created for the preset application module, these words are called second words, and they are included in the second vocabulary.

[0026] For example, words such as "sales promotion" and "insurance" are often set as abnormal words because they involve commercial promotion, but these words are allowed to appear in the insurance module of a certain financial website. Therefore, these words are included in the second vocabulary corresponding to the insurance module of a certain financial website.

[0027] S300: receiving at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in a preset application module, and matching a corresponding basic vocabulary from a plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category.

[0028] Different preset application modules have different reasons for setting abnormal words due to their different functions and application scenarios. Consequently, different vocabulary categories are prohibited from appearing in these preset application modules, and different basic vocabulary libraries are used to detect abnormal words in these preset application modules. Users select vocabulary categories from multiple vocabulary categories that are prohibited from appearing in the preset application modules. The server receives the vocabulary categories selected by the user and uses them to match the corresponding basic vocabulary library as the target basic vocabulary library. This target basic vocabulary library can then be used to detect abnormal words in the text to be tested in the preset application modules.

[0029] S400: Determine a vocabulary group corresponding to a preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary.

[0030] The server uses the target basic vocabulary, the first vocabulary and the second vocabulary to determine the vocabulary group corresponding to the preset application module, so that the vocabulary group includes all abnormal words set by the preset application module, as well as words that are allowed to appear in the preset application module but may be generally set as abnormal words by other application modules. The vocabulary group is used to detect abnormal words in the text to be detected in the preset application module.

[0031] When creating a vocabulary group for a preset application module, the target basic vocabulary is referenced. Thus, when creating a vocabulary group for a particular application module, only the vocabulary category needs to be selected. By referencing the basic vocabulary, the corresponding abnormal words can be included in the vocabulary group, effectively improving the efficiency of vocabulary group creation. In addition, the vocabulary groups of multiple application modules can reference the same basic vocabulary, avoiding repeated storage of the same vocabulary when creating vocabulary groups for multiple application modules, effectively reducing the storage space and storage cost of the vocabulary on the server side. By creating a first vocabulary for the preset application module, abnormal words not included in the target basic vocabulary can be supplemented, avoiding missing abnormal words when using the vocabulary group to detect abnormal vocabulary in the preset application module's to-be-detected text. By creating a second vocabulary for the preset application module, words included in the target basic vocabulary but not abnormal words for the preset application module can be listed, so that these words can be excluded when using the vocabulary group to detect abnormal vocabulary in the preset application module's to-be-detected text, which is conducive to improving the accuracy of abnormal vocabulary detection.

[0032] In some embodiments, when determining the vocabulary group, the server can determine the word repetition rate of the first vocabulary to remove duplicates from the first vocabulary when the word repetition rate is greater than a preset threshold.

[0033] S410: Compare the target basic vocabulary with the first vocabulary to obtain a vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary.

[0034] The server compares the target basic vocabulary with the first vocabulary, specifically comparing the basic vocabulary in the target basic vocabulary with the first vocabulary in the first vocabulary one by one, obtaining the repeated vocabulary and the number of repeated vocabulary in the target basic vocabulary and the first vocabulary, and calculating the vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary, which is equal to the number of repeated vocabulary divided by the number of first vocabulary.

[0035] In some embodiments, when comparing the target basic vocabulary and the first vocabulary, a lexical semantic comparison method can be used to calculate the semantic similarity between each word in the first vocabulary and each word in the target basic vocabulary. If the semantic similarity between a word in the first vocabulary and a word in the target basic vocabulary is greater than a similarity threshold, it indicates that the two words describe the same content, and the two words are identified as duplicate words.

[0036] S420: Determine whether the vocabulary repetition rate is greater than a preset threshold.

[0037] If so, executing S430: deleting the words that are repeated in the target basic vocabulary from the first vocabulary to obtain a deduplicated vocabulary.

[0038] The vocabulary repetition rate indicates the ratio of the number of repeated words that appear in both the first vocabulary and the target basic vocabulary to the total number of words in the first vocabulary. If the vocabulary repetition rate is greater than the preset threshold, it indicates that the number of repeated words included in both the first vocabulary and the target basic vocabulary is too large, that is, a large number of words are stored repeatedly, which will lead to a waste of storage resources. Therefore, these words are removed from the first vocabulary to effectively save storage space and reduce storage costs.

[0039] For example, the first vocabulary includes a total of 50 first words, and 15 of these 50 first words are also included in the target basic vocabulary. Therefore, the vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary is 30%, and the preset threshold is 10%. The vocabulary repetition rate is greater than the preset threshold, so these 15 words are removed from the first vocabulary, and the resulting deduplicated vocabulary includes a total of 35 first words.

[0040] S440: forming a vocabulary group corresponding to a preset application module according to the target basic vocabulary, the deduplication vocabulary and the second vocabulary.

[0041] The server uses the target basic vocabulary, the deduplication vocabulary, and the second vocabulary to form a vocabulary group corresponding to the preset application module. The target basic vocabulary and the deduplication vocabulary of this vocabulary group include all words that are not allowed to appear in the preset application module, and there are no duplicate words between the target basic vocabulary and the deduplication vocabulary. This effectively reduces storage costs while effectively improving the detection efficiency of abnormal vocabulary detection using the vocabulary group. In addition, the second vocabulary includes words that are allowed to appear in the preset application module, but may be included in the target basic vocabulary because they are generally set as abnormal words. These words can be excluded when using the vocabulary group for abnormal vocabulary detection, effectively reducing the possibility of misjudging the words in the text to be detected, which is conducive to improving the reliability and accuracy of vocabulary detection.

[0042] When the vocabulary repetition rate is less than or equal to the preset threshold, executing S450: retaining the vocabulary in the first vocabulary library that is repeated with the target basic vocabulary library.

[0043] S460: forming a vocabulary group corresponding to a preset application module according to the target basic vocabulary, the first vocabulary, and the second vocabulary.

[0044] When the vocabulary repetition rate is less than or equal to the preset threshold, it indicates that the number of repeated words included in both the first vocabulary and the target basic vocabulary is not large, and will not cause excessive waste of storage resources. In this case, there is no need to deduplicate the first vocabulary. The vocabulary group corresponding to the preset application module can be directly formed based on the target basic vocabulary, the first vocabulary and the second vocabulary, thereby effectively improving the efficiency of generating the vocabulary group.

[0045] In some embodiments, after the target basic vocabulary is updated, the vocabulary group of the preset application module may need to be regenerated. For details, see the following steps included in S400.

[0046] S470: Determine whether the target basic vocabulary is updated.

[0047] If so, execute S480: determine whether the update includes removing some basic words in the target basic vocabulary.

[0048] If so, executing S490: using the updated target basic vocabulary, the first vocabulary, and the second vocabulary to re-determine the vocabulary group corresponding to the preset application module.

[0049] After the repeated words in the first vocabulary and the target basic vocabulary are deleted from the first vocabulary, if the target basic vocabulary is updated, it indicates that the basic vocabulary in the target basic vocabulary has undergone certain changes. If some basic vocabulary is removed during the update, it may cause the words originally included in both the target basic vocabulary and the first vocabulary to be first deleted from the first vocabulary, and then removed from the target basic vocabulary due to the update of the target basic vocabulary, resulting in the word not being included in the first vocabulary and the target basic vocabulary, resulting in the word being omitted from the vocabulary group, which may lead to omission of abnormal vocabulary detection in the text to be detected. Therefore, it is necessary to generate the vocabulary group again based on the first vocabulary without deleting the repeated words to avoid omission of words, which is conducive to improving the reliability and accuracy of abnormal vocabulary detection using the vocabulary group.

[0050] For example, the target basic vocabulary and the first vocabulary originally included the word "insurance". After deduplication of the first vocabulary, the word "insurance" was not included in the deduplication vocabulary. After the target basic vocabulary was updated, the word "insurance" was removed from the target basic vocabulary. At this time, the abnormal vocabulary "insurance" originally belonging to the preset application module is not included in the target basic vocabulary and the deduplication vocabulary. This will result in abnormal vocabulary detection for the text to be detected of the preset application module. If the word "insurance" is in the file to be detected, the server cannot use the vocabulary group to detect the word, resulting in omission of abnormal vocabulary detection. Therefore, after the target basic vocabulary is updated to remove the vocabulary, it is necessary to use the updated target basic vocabulary, the first vocabulary and the second vocabulary to re-determine the vocabulary group corresponding to the preset application module to avoid omission of vocabulary.

[0051] In some embodiments, when using the updated target basic vocabulary, the first vocabulary and the second vocabulary to re-determine the vocabulary group corresponding to the preset application module, it is necessary to compare the updated target basic vocabulary with the first vocabulary to obtain the vocabulary repetition rate of the first vocabulary relative to the updated target basic vocabulary, and determine whether the vocabulary repetition rate is greater than a preset threshold. If so, delete the words that are repeated with the updated target basic vocabulary from the first vocabulary to obtain an updated deduplicated vocabulary. The vocabulary group corresponding to the preset application module is formed based on the updated target basic vocabulary, the updated deduplicated vocabulary and the second vocabulary. If the vocabulary repetition rate is less than or equal to the preset threshold, the vocabulary group corresponding to the preset application module is directly determined based on the updated target basic vocabulary, the first vocabulary and the second vocabulary.

[0052] S500: Receive a text to be detected input by a user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected. The abnormal vocabulary list includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module, and does not include second vocabulary that is allowed to appear in the preset application module.

[0053] Different preset application modules correspond to different vocabulary groups. After receiving the text to be detected from the preset application module input by the user, abnormal vocabulary detection can be performed on the text to be detected according to the vocabulary group corresponding to the preset application module to obtain an abnormal vocabulary list. The words in the abnormal vocabulary list are not allowed to appear in the preset application module.

[0054] In some embodiments, the target basic vocabulary and the first vocabulary in the vocabulary group corresponding to the preset application module are used to mark the words in the text to be detected, and then the marked words are matched using the second vocabulary. For details, see the following steps included in S500.

[0055] S510: Mark the words that match the basic words and the first words in the word library group corresponding to the preset application module in the text to be detected, and obtain a marked word list.

[0056] The server divides the text to be detected into words to obtain multiple words to be detected included in the text to be detected, uses the basic words in the target basic word library of the word library group and the first word in the first word library to compare with the multiple words to be detected in turn, and marks the words that match the basic words and the first words to obtain a marked word list, which includes all the basic words and the first words that appear in the text to be detected.

[0057] S520: Deleting words that match the second word in the vocabulary group corresponding to the preset application module from the marked word list to obtain an abnormal word list.

[0058] The server compares the second word in the second vocabulary with the words in the marked word list to delete the words in the marked word list that match the second word, thereby obtaining an abnormal word list. The abnormal word list includes all abnormal words in the text to be detected, and does not include words that are allowed to appear, thereby accurately and efficiently detecting abnormal words in the text to be detected.

[0059] In some embodiments, the target basic vocabulary, the first vocabulary, and the second vocabulary in the vocabulary group corresponding to the preset application module are used to detect the vocabulary in the text to be detected, and then the list obtained by comparison is processed to obtain a list of abnormal vocabulary. For details, see the following steps included in S500.

[0060] S530: Mark the words matching the first word, the second word and the basic word in the vocabulary group corresponding to the preset application module in the text to be detected, and obtain a first word list, a second word list and a basic word list.

[0061] The server divides the text to be detected into words to obtain multiple words to be detected included in the text to be detected, uses the basic words in the target basic vocabulary, the first words in the first vocabulary, and the second words in the second vocabulary to compare with the multiple words to be detected, and marks the words that match the basic words, the first words, and the second words, respectively, to obtain a first vocabulary list, a second vocabulary list, and a basic vocabulary list.

[0062] S540: Combining the first vocabulary list and the basic vocabulary list to form a marked vocabulary list, deleting words that match the second words in the second vocabulary list from the marked vocabulary list to obtain an abnormal vocabulary list.

[0063] By combining the first vocabulary list and the basic vocabulary list to form a marked vocabulary list, the marked vocabulary list includes all basic vocabulary and the first vocabulary that appear in the text to be tested. In other words, the marked vocabulary list includes all abnormal vocabulary in the text to be tested. Then, the words in the marked vocabulary list that match the second vocabulary in the second vocabulary list are deleted. In other words, the words allowed to appear in the preset application module are removed from the marked vocabulary list, effectively improving the accuracy of abnormal vocabulary detection.

[0064] By creating basic vocabulary libraries of different vocabulary categories, the basic vocabulary in the basic vocabulary libraries are all vocabulary that are generally set as abnormal words, so that the same basic vocabulary library can be referenced when determining vocabulary library groups for different application modules, effectively reducing the possibility of repeated vocabulary storage, and effectively reducing the storage space and storage cost of the vocabulary library on the server side. Secondly, by creating a first vocabulary library and a second vocabulary library for the preset application module, the first vocabulary in the first vocabulary library is the vocabulary set as an abnormal word in the preset application module, and the second vocabulary in the second vocabulary library is the vocabulary that is often set as an abnormal word but is not set as an abnormal word in the preset application module. Using the target basic vocabulary library, the first vocabulary library and the second vocabulary library A word library and a second word library determine a word library group corresponding to a preset application module, so that the word library group includes abnormal words of the preset application module, so that abnormal vocabulary detection can be performed on the text to be detected in the preset application module, and includes words that may appear in the basic word library but are not abnormal words in the preset application module, so as to improve the accuracy of abnormal vocabulary detection and avoid misjudging words that are abnormal words that are not in this application module, so that the word library group can meet the personalized needs of abnormal vocabulary detection on the text to be detected in the preset application module, and further provide differentiated word library groups for different application modules to meet the detection needs of different application modules.

[0065] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0066] In one embodiment, a device for detecting abnormal words is provided, and the device for detecting abnormal words corresponds to the method for detecting abnormal words in the above embodiment. Figure 3 As shown, the abnormal vocabulary detection device includes a generation module 101, a creation module 102, a matching module 103, and a detection module 104. The detailed description of each functional module is as follows:

[0067] A generating module 101 is configured to read a vocabulary data source, classify the vocabulary in the vocabulary data source according to vocabulary categories, and generate a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category;

[0068] A creation module 102 is configured to create a first vocabulary and a second vocabulary for a preset application module, wherein the first vocabulary includes a plurality of first words that are not allowed to appear in the preset application module, and the second vocabulary includes a plurality of second words that are allowed to appear in the preset application module;

[0069] a matching module 103 configured to receive at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not permitted to appear in a preset application module, match a corresponding basic vocabulary from a plurality of basic vocabulary libraries according to the vocabulary category as a target basic vocabulary library, and determine a vocabulary library group corresponding to the preset application module using the target basic vocabulary library, the first vocabulary library, and the second vocabulary library;

[0070] The detection module 104 is used to receive the text to be detected input by the user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected. The abnormal vocabulary list includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module, and does not include second vocabulary that is allowed to appear in the preset application module.

[0071] In one embodiment, when the matching module 103 uses the target basic vocabulary, the first vocabulary, and the second vocabulary to determine the vocabulary group corresponding to the preset application module, it is configured to:

[0072] Comparing the target basic vocabulary with the first vocabulary to obtain a vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary;

[0073] Determine whether the vocabulary repetition rate is greater than a preset threshold;

[0074] If so, delete the words that are repeated in the target basic vocabulary from the first vocabulary to obtain a deduplicated vocabulary;

[0075] A vocabulary group corresponding to a preset application module is formed according to the target basic vocabulary, the deduplication vocabulary and the second vocabulary;

[0076] When the vocabulary repetition rate is less than or equal to a preset threshold, a vocabulary group corresponding to the preset application module is directly formed according to the target basic vocabulary, the first vocabulary and the second vocabulary.

[0077] In one embodiment, when the matching module 103 compares the target basic vocabulary with the first vocabulary to obtain the vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary, it is configured to:

[0078] Comparing the target basic vocabulary with the first vocabulary to obtain the repeated words and their numbers between the target basic vocabulary and the first vocabulary;

[0079] The vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary is calculated, where the vocabulary repetition rate is equal to the number of repeated words divided by the number of first words.

[0080] In one embodiment, after determining the vocabulary group corresponding to the preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary, the matching module 103 is further configured to:

[0081] Determine whether the target basic vocabulary is updated;

[0082] If so, determining whether the update includes removing some basic words in the target basic vocabulary;

[0083] If so, the updated target basic vocabulary, the first vocabulary, and the second vocabulary are used to determine the vocabulary group corresponding to the preset application module again.

[0084] In one embodiment, the detection module 104, when performing vocabulary detection on the text to be detected based on the vocabulary group corresponding to the preset application module to obtain a list of abnormal vocabulary included in the text to be detected, is configured to:

[0085] Marking the words that match the basic words and the first words in the vocabulary group corresponding to the preset application module in the text to be detected to obtain a marked word list;

[0086] The words that match the second word in the word library group corresponding to the preset application module are deleted from the marked word list to obtain an abnormal word list.

[0087] In one embodiment, the detection module 104, when performing vocabulary detection on the text to be detected based on the vocabulary group corresponding to the preset application module to obtain a list of abnormal vocabulary included in the text to be detected, is configured to:

[0088] Marking the words that match the first words, the second words, and the basic words in the vocabulary group corresponding to the preset application module in the text to be detected, respectively, to obtain a first word list, a second word list, and a basic word list;

[0089] The first vocabulary list and the basic vocabulary list are combined to form a marked vocabulary list, and words that match the second vocabulary in the second vocabulary list are deleted from the marked vocabulary list to obtain an abnormal vocabulary list.

[0090] The present invention provides a device for detecting abnormal words. By creating basic word libraries of different word categories, the basic words in the basic word library are all words that are generally set as abnormal words, so that the same basic word library can be referenced when determining word library groups for different application modules, effectively reducing the possibility of repeated word storage, and effectively reducing the storage space and storage cost of the word library on the server side. Secondly, by creating a first word library and a second word library for a preset application module, the first word in the first word library is a word set as an abnormal word in the preset application module, and the second word in the second word library is a word that is often set as an abnormal word but is not set as an abnormal word in the preset application module, The target basic vocabulary, the first vocabulary and the second vocabulary are used to determine the vocabulary group corresponding to the preset application module, so that the vocabulary group includes abnormal words of the preset application module, so that abnormal vocabulary detection can be performed on the text to be detected in the preset application module, and includes words that may appear in the basic vocabulary but are not abnormal words in the preset application module, so as to improve the accuracy of abnormal vocabulary detection and avoid misjudging abnormal words that are not in this application module, so that the vocabulary group can meet the personalized needs of abnormal vocabulary detection on the text to be detected in the preset application module, and then provide differentiated vocabulary groups for different application modules to meet the detection needs of different application modules.

[0091] For the specific definition of the abnormal vocabulary detection device, please refer to the definition of the abnormal vocabulary detection method above, which will not be repeated here. The various modules in the above-mentioned abnormal vocabulary detection device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0092] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side method for detecting abnormal vocabulary.

[0093] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the client side of a method for detecting abnormal vocabulary.

[0094] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0095] Reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category;

[0096] Creating a first vocabulary library and a second vocabulary library for the preset application module, wherein the first vocabulary library includes a plurality of first words that are not allowed to appear in the preset application module, and the second vocabulary library includes a plurality of second words that are allowed to appear in the preset application module;

[0097] receiving at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in a preset application module, and matching a corresponding basic vocabulary from a plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category;

[0098] Determine a vocabulary group corresponding to a preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary;

[0099] Receive the text to be detected input by the user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal words included in the text to be detected. The abnormal word list includes basic words and first words that are not allowed to appear in the preset application module.

[0100] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0101] Reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category;

[0102] Creating a first vocabulary library and a second vocabulary library for the preset application module, wherein the first vocabulary library includes a plurality of first words that are not allowed to appear in the preset application module, and the second vocabulary library includes a plurality of second words that are allowed to appear in the preset application module;

[0103] receiving at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in a preset application module, and matching a corresponding basic vocabulary from a plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category;

[0104] Determine a vocabulary group corresponding to a preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary;

[0105] Receive the text to be detected input by the user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal words included in the text to be detected. The abnormal word list includes basic words and first words that are not allowed to appear in the preset application module.

[0106] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant description of the server side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0107] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0108] Among them, any reference to memory, storage, database or other media used in the various embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0109] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0110] It should be noted that if software tools or components other than those of our company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

[0111] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for detecting abnormal words, characterized in that: The detection method comprises: Reading a vocabulary data source, classifying the vocabulary in the vocabulary data source according to vocabulary categories, and generating a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category; Creating a first vocabulary library and a second vocabulary library for a preset application module, wherein the first vocabulary library includes a plurality of first words that are not allowed to appear in the preset application module, and the second vocabulary library includes a plurality of second words that are allowed to appear in the preset application module; receiving at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in the preset application module, and matching a corresponding basic vocabulary from the plurality of basic vocabulary libraries as a target basic vocabulary library according to the vocabulary category; Determining a vocabulary group corresponding to the preset application module using the target basic vocabulary, the first vocabulary, and the second vocabulary; receiving a text to be detected input by a user, performing vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtaining a list of abnormal vocabulary included in the text to be detected, wherein the list of abnormal vocabulary includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module; The method of performing vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module to obtain a list of abnormal vocabulary included in the text to be detected includes: marking the vocabulary that matches the basic vocabulary and the first vocabulary in the vocabulary group corresponding to the preset application module in the text to be detected to obtain a marked vocabulary list, and deleting the vocabulary that matches the second vocabulary in the vocabulary group corresponding to the preset application module from the marked vocabulary list to obtain the abnormal vocabulary list; or The performing vocabulary detection on the to-be-detected text according to the vocabulary group corresponding to the preset application module to obtain a list of abnormal vocabulary included in the to-be-detected text includes: marking, in the to-be-detected text, vocabulary that matches the first vocabulary, the second vocabulary, and the basic vocabulary in the vocabulary group corresponding to the preset application module, respectively, to obtain a first vocabulary list, a second vocabulary list, and a basic vocabulary list; The first vocabulary list and the basic vocabulary list are combined to form a marked vocabulary list, and words matching the second vocabulary in the second vocabulary list are deleted from the marked vocabulary list to obtain the abnormal vocabulary list.

2. The method for detecting abnormal words according to claim 1, wherein: The determining of the vocabulary group corresponding to the preset application module by using the target basic vocabulary, the first vocabulary, and the second vocabulary includes: Comparing the target basic vocabulary with the first vocabulary to obtain a vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary; Determining whether the vocabulary repetition rate is greater than a preset threshold; If yes, deleting the words that are repeated in the target basic vocabulary from the first vocabulary to obtain a deduplicated vocabulary; A vocabulary group corresponding to the preset application module is formed according to the target basic vocabulary, the deduplication vocabulary and the second vocabulary.

3. The method for detecting abnormal words according to claim 2, wherein: The comparing the target basic vocabulary with the first vocabulary to obtain a vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary includes: Comparing the target basic vocabulary with the first vocabulary to obtain the repeated words and the number of repeated words between the target basic vocabulary and the first vocabulary; The vocabulary repetition rate of the first vocabulary relative to the target basic vocabulary is calculated, where the vocabulary repetition rate is equal to the number of the repeated words divided by the number of the first words.

4. The method for detecting abnormal words according to claim 2, wherein: After determining whether the vocabulary repetition rate is greater than a preset threshold, the method further includes: When the vocabulary repetition rate is less than or equal to the preset threshold, retaining the vocabulary in the first vocabulary library that is repeated with the target basic vocabulary library; A vocabulary group corresponding to the preset application module is formed according to the target basic vocabulary, the first vocabulary and the second vocabulary.

5. The method for detecting abnormal vocabulary according to claim 1 or 2, wherein: The detection method further comprises: Determining whether the target basic vocabulary is updated; If so, determining whether the updating includes removing some basic words in the target basic vocabulary; If so, the updated target basic vocabulary, the first vocabulary, and the second vocabulary are used to determine the vocabulary group corresponding to the preset application module again.

6. A device for detecting abnormal words, for implementing the method for detecting abnormal words according to any one of claims 1 to 5, characterized in that: include: A generating module, configured to read a vocabulary data source, classify the vocabulary in the vocabulary data source according to vocabulary categories, and generate a plurality of basic vocabulary libraries of different vocabulary categories, each basic vocabulary library including a plurality of basic vocabulary of the same vocabulary category; a creation module, configured to create a first vocabulary library and a second vocabulary library for a preset application module, wherein the first vocabulary library includes a plurality of first words that are not allowed to appear in the preset application module, and the second vocabulary library includes a plurality of second words that are allowed to appear in the preset application module; a matching module, configured to receive at least one vocabulary category selected by a user from a plurality of vocabulary categories that is not allowed to appear in the preset application module, match a corresponding basic vocabulary from the plurality of basic vocabulary libraries as a target basic vocabulary library based on the vocabulary category, and determine a vocabulary library group corresponding to the preset application module using the target basic vocabulary library, the first vocabulary library, and the second vocabulary library; The detection module is used to receive a text to be detected input by a user, perform vocabulary detection on the text to be detected according to the vocabulary group corresponding to the preset application module, and obtain a list of abnormal vocabulary included in the text to be detected, wherein the list of abnormal vocabulary includes basic vocabulary and first vocabulary that are not allowed to appear in the preset application module, and does not include second vocabulary that is allowed to appear in the preset application module.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the abnormal vocabulary detection method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the abnormal vocabulary detection method according to any one of claims 1 to 5 are implemented.

Images