Text processing method and device
Through the pre-processing and iterative calculation of text processing methods, combined with standard vocabulary list and filtering rules, the problems of high manpower consumption and high model maintenance costs in the expansion and detection of illegal word sets in the existing technology are solved, and efficient and accurate illegal word Filter and maintain.
Patent Information
- Application Number
- CN202411103281.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-08-11
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-19
AI Technical Summary
The prior art has problems such as high labor consumption, long time consumption and high model maintenance costs in the expansion and detection of violation word sets, especially in the preprocessing and optimization stages.
Provide a text processing method, which preprocesses, sentences and word segments by obtaining the pending text, uses preset word replacement algorithms such as BPE to iterate the word set, determine the violation words, and add them to the violation words in the target field. library, combining standard vocabulary lists and filtering rules to improve accuracy and efficiency.
It realizes the accuracy of efficient screening of illegal words in unsupervised scenarios, saves manpower and material resources, simplifies the determination and maintenance process of illegal words, and is suitable for detection and extraction of illegal words in multiple fields.
Smart Images

Figure CN118982022A_ABST
Abstract
Description
[0001] This disclosure claims the priority of the Chinese patent application filed with the Chinese Patent Office on August 11, 2023, with application number 202311015347.9 and invention name “Text Processing Method and Device”, the entire contents of which are incorporated by reference in this disclosure. Technical Field
[0002] The present application relates to the field of computer technology, and in particular to a text processing method and device. Background Art
[0003] In the prior art, in the scenario of expanding the set of illegal words, the methods usually adopted are manual summary and / or machine learning methods; manual summary of illegal keywords not only consumes a large amount of human resources, but also tests the language summary ability of the auditor. If the summary is not detailed enough, it will lead to more redundant data in the vocabulary; if it is too detailed, it will take a lot of time. Although the machine learning method can save human resources and improve the efficiency of expanding the set of illegal words, the preprocessing stage still requires a large amount of corpus to train the model, and the trained model needs to be continuously optimized, and the maintenance cost is high. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention
[0004] In view of this, the embodiment of the present application provides a text processing method to solve the technical defects existing in the prior art. The embodiment of the present application also provides a text processing device, a computing device, and a computer-readable storage medium.
[0005] According to a first aspect of an embodiment of the present application, a text processing method is provided, comprising:
[0006] Get the text to be processed;
[0007] Preprocessing the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit;
[0008] The word set corresponding to each sentence unit is iteratively calculated using a preset word replacement algorithm, and the illegal words in the text to be processed are determined according to the iterative calculation results.
[0009] Optionally, the preprocessing of the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit includes:
[0010] Perform sentence processing on the text to be processed to obtain at least one sentence unit corresponding to the text to be processed;
[0011] Each sentence unit is segmented to obtain the word set corresponding to each sentence unit.
[0012] Optionally, determining the illegal words in the text to be processed according to the iterative calculation result includes:
[0013] Determine the remaining word units in the word set corresponding to each sentence unit according to the iteration result;
[0014] The remaining word units in the word set corresponding to each sentence unit are regarded as the illegal words.
[0015] Optionally, before the step of iteratively calculating the word set corresponding to each sentence unit using a preset word replacement algorithm is performed, the method further includes:
[0016] Match the word set corresponding to each sentence unit with a preset standard word list;
[0017] Selecting a word set that does not match the standard word list as a target word set;
[0018] The iterative calculation of the word set corresponding to each sentence unit by using a preset word replacement algorithm includes:
[0019] The target word set is iteratively calculated using a preset word replacement algorithm.
[0020] Optionally, determining the illegal words in the text to be processed according to the iterative calculation result includes:
[0021] Determining the remaining word units in the target word set according to the iteration result;
[0022] The remaining word units in the target word set are matched with the standard word list, and unmatched word units are selected as the offending words.
[0023] Optionally, obtaining the text to be processed includes:
[0024] Acquire the text to be processed corresponding to the target field;
[0025] Wherein, after the step of determining the illegal words in the text to be processed according to the iterative calculation result is executed, the method further includes:
[0026] The illegal words are added to the illegal word library corresponding to the target domain.
[0027] Optionally, before the step of iteratively calculating the word set corresponding to each sentence unit using a preset word replacement algorithm is performed, the method further includes:
[0028] Filtering an initial word set containing character units from the word set corresponding to each sentence unit;
[0029] splicing the character units contained in the initial word set according to a preset splicing strategy to obtain a word set to be processed;
[0030] The iterative calculation of the word set corresponding to each sentence unit by using a preset word replacement algorithm includes:
[0031] The preset word replacement algorithm is used to iteratively calculate the word set to be processed.
[0032] Optionally, the step of splicing the character units contained in the initial word set according to a preset splicing strategy to obtain a word set to be processed includes:
[0033] Determine the frequencies corresponding to the character units included in the initial word set, and the adjacent character units associated with the character units;
[0034] The character unit and the adjacent character unit are spliced according to the frequency, and the to-be-processed word set is obtained according to the splicing result.
[0035] Optionally, after the step of determining the illegal words in the text to be processed according to the iterative calculation result is performed, the method further includes:
[0036] Filter the illegal words according to the preset filtering rules, obtain the target illegal words and write them into the target illegal word library;
[0037] When the text to be desensitized is received, the text to be desensitized is processed based on the target violation library to obtain the target text.
[0038] According to a second aspect of an embodiment of the present application, there is provided a text processing device, including:
[0039] An acquisition module, configured to acquire text to be processed;
[0040] A processing module is configured to pre-process the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit;
[0041] The determination module is configured to perform iterative calculation on the word set corresponding to each sentence unit using a preset word replacement algorithm, and determine the illegal words in the text to be processed according to the iterative calculation result.
[0042] According to a third aspect of an embodiment of the present application, a computing device is provided, including:
[0043] Memory and processor;
[0044] The memory is used to store computer-executable instructions, and the processor implements the steps of the text processing method when executing the computer-executable instructions.
[0045] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the text processing method are implemented.
[0046] According to a fifth aspect of an embodiment of the present application, a chip is provided, which stores a computer program, and the computer program implements the steps of the text processing method when executed by the chip.
[0047] The text processing method provided by the present application can realize that when it is necessary to filter out illegal words, after obtaining the text to be processed, the text to be processed can be pre-processed to obtain sentence units corresponding to the text to be processed and word sets corresponding to each sentence unit, and then a preset word replacement algorithm is used to iteratively calculate the word set corresponding to each sentence unit, so as to obtain illegal words from the text to be processed. This can not only save manpower and material resources, but also ensure the accuracy of illegal word filtering, and realize the determination of illegal words in an unsupervised scenario, so as to facilitate the downstream to perform subsequent processing operations based on the illegal words obtained by screening. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a flowchart of a text processing method provided by an embodiment of the present application;
[0049] Figure 2 is a schematic diagram of a text processing method provided by an embodiment of the present application;
[0050] Figure 3 is a processing flow chart of another text processing method provided by an embodiment of the present application;
[0051] Figure 4 is a structural schematic diagram of a text processing device provided by an embodiment of the present application;
[0052] Figure 5 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0053] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.
[0054] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.
[0055] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.
[0056] First, the terms involved in one or more embodiments of the present invention are explained.
[0057] BPE algorithm: (byte pair encoder), byte pair encoding, also known as digram coding, is mainly used for data compression. The algorithm is described as an iterative process in which the most common pair of characters in a string is replaced by a character that does not appear in this string.
[0058] Unsupervised learning: Due to the lack of sufficient prior knowledge, it is difficult to manually label the categories or the cost of manual category labeling is too high. Therefore, it is hoped that computers can complete these tasks on behalf of humans, or at least provide some help. Therefore, various problems in pattern recognition can be solved based on training samples with unknown categories (not labeled), which is called unsupervised learning.
[0059] Sentence division: Sentence division is performed according to preset rules and based on punctuation marks.
[0060] Word segmentation: It is the process of recombining continuous character sequences into word sequences according to certain specifications.
[0061] Violation words: word units whose semantic expressions do not match the representation of the target domain, or word units that do not match the representation rules.
[0062] Violation word set: a set used to store violation words corresponding to the target domain, and the violation words in the violation word set are relative to the target domain.
[0063] In the present application, a text processing method is provided. The present application also relates to a text processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0064] Figure 1 A flowchart of a text processing method provided according to an embodiment of the present application is shown, which specifically includes the following steps:
[0065] Step S102, obtaining the text to be processed.
[0066] Specifically, the text to be processed refers to the corpus containing illegal words; in the illegal word detection scenario, articles to be published, emails to be sent, or chat content to be sent can all be used as text to be processed for subsequent illegal word detection; or in the illegal word extraction scenario, such as the expansion of the illegal word set, sample construction for model training, etc., the text involved can be used as text to be processed for subsequent extraction of illegal words and handed over to downstream businesses for use.
[0067] It should be noted that the application approaches after detecting illegal words in different scenarios are different. In the process of determining illegal words, the same or corresponding description content of this embodiment can be referred to. For the convenience of description, this embodiment takes the extraction of illegal words and the expansion of the illegal word set as an example for explanation. The illegal word determination process of other scenarios will not be described in detail here.
[0068] In the specific implementation, considering that the definitions of illegal words in different fields are different, for example, word unit S is defined as an illegal word in field A, while the word unit is defined as a normal word unit in field B, if illegal words are not screened according to field division, the problem of inaccurate positioning of illegal words may occur. Therefore, when the processing text is obtained, in order to ensure the accuracy of determining illegal words when locating illegal words later, processing can be performed based on the field. In this embodiment, the specific implementation method is as follows:
[0069] The to-be-processed text corresponding to the target domain is obtained; after determining the illegal words contained in the to-be-processed text, the illegal words are added to the illegal word library corresponding to the target domain.
[0070] Specifically, the illegal word library specifically refers to a database corresponding to the target field that stores illegal words in the field, and the illegal words contained in the illegal word library are all word units that are illegal words relative to the target field. Based on this, after obtaining the text to be processed in the target field, the illegal words will be removed in accordance with the set rules corresponding to the field, and the illegal words determined from the text to be processed will be written into the illegal word library corresponding to the target field, so that the downstream business of the corresponding target field can use the illegal words in the illegal word library for subsequent processing operations, such as text desensitization processing, model training processing, etc. The specific application scenario can be set according to the actual application scenario, and this embodiment does not make too many restrictions here.
[0071] In summary, by obtaining the to-be-processed text of the corresponding target field and storing the determined illegal words in the illegal word library of the corresponding target field, it can not only ensure that the determination of illegal words is related to the target field, but also the accuracy is high enough to facilitate the use of downstream businesses in the target field.
[0072] Step S104, preprocessing the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit.
[0073] Specifically, after the text to be processed is obtained as described above, in order to ensure that the offending words can be successfully obtained from the text to be processed later, it is also necessary to preprocess the text to be processed in the data preparation stage to obtain at least one sentence unit contained in the text to be processed and a word set corresponding to each sentence unit.
[0074] Among them, preprocessing the text to be processed refers to sentence and word segmentation; accordingly, the sentence unit specifically refers to the sentence unit obtained after the text to be processed is segmented; accordingly, the word set specifically refers to the set composed of word units contained in each sentence unit, and each word unit contains at least one character.
[0075] Furthermore, the preprocessing of the text to be processed is implemented as steps S1042 to S1044.
[0076] Step S1042, performing sentence processing on the text to be processed to obtain at least one sentence unit corresponding to the text to be processed;
[0077] Step S1044, performing word segmentation processing on each sentence unit to obtain a word set corresponding to each sentence unit.
[0078] Specifically, sentence processing specifically refers to dividing the text to be processed into multiple sentence units, wherein sentence processing can be implemented using sentence processing rules, such as setting sentence segmentation based on punctuation marks, or using a sentence model. This embodiment does not make any restrictions here; correspondingly, word segmentation processing specifically refers to dividing the word units contained in each sentence unit, wherein word segmentation processing can use the jieba word segmentation tool, the HanLP word segmentation tool, or the FudanNLP word segmentation tool, etc. During specific implementation, it can be selected according to actual needs, and this embodiment does not make any restrictions here.
[0079] Based on this, after obtaining the text to be processed, in order to facilitate the subsequent screening of illegal words based on sentence units, the text to be processed can be first segmented into sentences to obtain at least one sentence unit corresponding to the text to be processed, and then each sentence unit can be segmented to obtain a word set corresponding to each sentence unit. Subsequently, the sentence unit containing the illegal words can be determined by screening each word set, and the illegal words can be further determined to obtain the illegal words in the text to be processed.
[0080] For example, obtain the text to be processed in the corresponding football field {… Although player A has participated in many leagues, his ball kick is really bad...}, and use punctuation marks as segmentation units to segment the text to be processed into sentences, and obtain the first sentence unit {Although player A has participated in many leagues} and the second sentence unit {But his ball kick is really bad}; then use the jieba word segmentation tool to segment each sentence unit separately, and obtain the word set S1 corresponding to the first sentence unit as {Player, A, although, participated, in, many, leagues}, and the word set S2 corresponding to the second sentence unit is {But, his, ball, kicks, is really bad}, so as to facilitate the subsequent determination of illegal words based on word sets.
[0081] In summary, by performing sentence and word segmentation processing on the text to be processed in the preprocessing stage, the word set corresponding to each sentence unit can be accurately constructed, which facilitates the subsequent use of the preset word replacement algorithm to accurately determine the illegal words in the word set.
[0082] Step S106, using a preset word replacement algorithm to iteratively calculate the word set corresponding to each sentence unit, and determining the illegal words in the text to be processed according to the iterative calculation result.
[0083] Specifically, after obtaining the word set corresponding to each sentence unit, further, in order to be able to filter out illegal words from the word set, a preset word replacement algorithm can be used to iteratively calculate the word units contained in the word set corresponding to each sentence unit, so as to filter out illegal words in the text to be processed according to the iteration result. Among them, the preset word replacement algorithm specifically refers to the BPE algorithm, which iteratively replaces the word units in the word set through the BPE algorithm, and finally uses the remaining word units as the illegal word set.
[0084] Furthermore, when each word set is iteratively calculated based on a preset word replacement algorithm, the word unit with the highest new frequency is actually selected for replacement, and then a new word set is formed. This is repeated until no replacement can be performed. The remaining word units can be regarded as illegal words. In this embodiment, the specific implementation is as steps S1062 to S1064.
[0085] Step S1062, determining the remaining word units in the word set corresponding to each sentence unit according to the iteration result;
[0086] Step S1064: The remaining word units in the word set corresponding to each sentence unit are used as the illegal words.
[0087] Specifically, the remaining word units specifically refer to the word units that cannot be replaced after the iterative replacement process of the word set corresponding to each sentence unit. Based on this, the word units contained in each word set are continuously iteratively replaced by a preset word replacement algorithm until they cannot be replaced. Then, the remaining word units in the word set corresponding to each sentence unit can be determined according to the iteration result, and finally used as the illegal words corresponding to the text to be processed.
[0088] In other words, the BPE algorithm can be used directly to determine the illegal words, that is, replace each word unit in the word set according to the frequency of occurrence in the text to be processed, and give priority to replacing the words with the highest frequency; through iterative replacement, determine that the remaining word units in the sentence unit are illegal words. For example, the word units in the word set S1-1{qw, er, ty, ui, op} are directly replaced by the BPE algorithm, and the remaining word units are "qw" and "er"; similarly, the word units in the word set S2-2{asd, fg, hj, kl} are replaced, and the remaining word units are "asd" and "fg"; at this time, "qw", "er", "asd" and "fg" can be regarded as illegal words.
[0089] Continuing with the above example, after obtaining the word set S1 corresponding to the first sentence unit {Player A, although, participated, in, many, leagues}, and the word set S2 corresponding to the second sentence unit {But, the ball, he, kicked, really stinks}, use the BPE algorithm to replace each word unit in the word set, and after each replacement, replace the word units in the new word set again until no further replacement is possible, and it is determined that all the word units in the word set S1 corresponding to the first sentence unit have been replaced, and the remaining word unit in the word set S2 corresponding to the second sentence unit is "really stinks". At this time, the word unit "really stinks" can be used as an illegal word in the text to be processed, so as to be written into the corresponding illegal vocabulary in the football field for subsequent use.
[0090] In summary, by adopting the preset word replacement algorithm to determine the illegal words, not only can the accuracy of illegal word determination be improved, but also the omission of illegal words can be avoided, thereby accurately extracting illegal words from the text to be processed for downstream business use.
[0091] In addition, when filtering out illegal words from the text to be processed, considering that the word set of the sentence unit corresponding to the text to be processed contains a large number of word units, if the illegal words are determined by traversal, it will not only consume a lot of time, but also the accuracy of determining illegal words will be at risk. Therefore, in order to improve the efficiency and accuracy of determining illegal words, a preset standard word list can be used to perform preliminary filtering first, and then determine the illegal words based on the filtering results. In this embodiment, the specific implementation is as shown in steps S1162 to S1166.
[0092] Step S1162, matching the word set corresponding to each sentence unit with a preset standard word list;
[0093] Step S1164, selecting a word set that does not match the standard word list to be determined as a target word set;
[0094] Step S1166: Iteratively calculate the target word set using a preset word replacement algorithm.
[0095] Specifically, the preset standard word list specifically refers to a word list composed of non-violation word units. By comparing the word set corresponding to each sentence unit with the word list, the sentence unit containing the violation word can be preliminarily determined, that is, the word set corresponding to the sentence unit containing the violation word is the target word, so that the violation word can be determined from the sentence unit containing the violation word in the subsequent process.
[0096] Based on this, firstly, the word units contained in the word set corresponding to each sentence unit are matched with the preset standard word list. If the match is successful, it means that the word unit is a non-illegal word. If the match is unsuccessful, it means that the word unit is an illegal word. Secondly, the word set corresponding to the unmatched word unit can be determined as the target word set according to the matching result. Finally, the preset word replacement algorithm is used to iteratively calculate the word units contained in the target word set, and the illegal word can be determined according to the iterative calculation result.
[0097] Among them, the process of iteratively calculating the target word set using the preset word replacement algorithm is similar to the above-mentioned iterative calculation process of the word set corresponding to each sentence unit. For the convenience of description in this embodiment, the same description content in the above-mentioned implementation method can be referred to, and no further details will be given here.
[0098] In summary, by using a preset standard word list to filter out word sets containing non-violation words, only retaining word sets that may contain violation words as target word sets for subsequent violation word determination processing, the impact of redundant data can be effectively reduced, thereby improving the efficiency of violation word determination.
[0099] Furthermore, after determining the target word set, although the illegal words in the target word set can be determined through iterative processing, the accuracy cannot be guaranteed. Therefore, the iterative results can be filtered again in combination with the standard word list to accurately determine the illegal words in the text to be processed. In this embodiment, the specific implementation is as follows: steps S11662 to S1164.
[0100] Step S11662, determining the remaining word units in the target word set according to the iteration result;
[0101] Step S11664, matching the remaining word units in the target word set with the standard word list, and selecting unmatched word units as the offending words.
[0102] Specifically, the remaining word units in the target word set specifically refer to word units that cannot be replaced by the word replacement algorithm, and there is a risk of violation of the rules. Based on this, after the target word set that may contain illegal words is screened out through the standard word list, in order to further improve the accuracy of determining illegal words, the preset word replacement algorithm can be used to iteratively calculate the target word set, and the remaining word units in the target word set are determined according to the calculation results. After that, the remaining word units are matched with the standard word list again, and the unmatched ones are selected as the illegal words of the text to be processed.
[0103] That is to say, since the standard vocabulary is not considered in the iterative replacement process, after determining the offending words, the offending words can be compared with the standard vocabulary, and then the word units that appear in the standard vocabulary in the offending words can be filtered out, and the remaining ones can be used as offending words. For example, after determining that "qw", "er", "asd" and "fg" are the remaining word units, each word unit is compared with the standard vocabulary one by one, and it is determined that "qw" and "fg" both exist in the standard vocabulary, then "qw" and "fg" are removed from the remaining word units, and the remaining word units "er" and "asd" are the offending words.
[0104] Using the above example, determine the standard vocabulary corresponding to the football field, then match the word units contained in word set S1 and word set S2 with the standard vocabulary, and determine that the word unit contained in S1 matches the standard vocabulary according to the matching results, and there are word units in word set S2 that do not match the standard vocabulary, then the word set S2 corresponding to the second sentence unit is used as the target word set. Further, use the BPE algorithm to iteratively calculate the word units in word set S2, and determine that the remaining word units in word set S2 are "kicked" and "really smelly"; determine that both have the risk of violating the rules, and finally match the word units "kicked" and "really smelly" with the standard vocabulary again, and determine that "really smelly" is not successfully matched, then select the word unit "really smelly" as the illegal word of the text to be processed, so as to be written into the illegal vocabulary corresponding to the football field for subsequent use.
[0105] In summary, after determining the target word set that may contain illegal words through the standard word list, in order to further ensure the accuracy of illegal word determination, after the iteration of the target word set is completed, a secondary screening can be performed through the standard word list to filter out illegal words from the remaining word units, thereby ensuring the accuracy of illegal word determination while avoiding omission of illegal words.
[0106] In addition, considering that after word segmentation, each word set may contain word units consisting of characters, if the illegal words are determined on this basis, the characters may also be regarded as illegal words, and the accuracy of illegal word determination cannot be guaranteed. Therefore, when there are word units consisting of characters, you can choose to form new word units by splicing, and then determine the illegal words. In this embodiment, the specific implementation is as shown in steps S1262 to S1266.
[0107] Step S1262, screening the initial word set containing the character unit from the word set corresponding to each sentence unit.
[0108] Step S1264, splicing the character units contained in the initial word set according to a preset splicing strategy to obtain a word set to be processed.
[0109] Specifically, the initial word set specifically refers to the word set corresponding to each sentence unit, which contains word units in the form of character units; correspondingly, the splicing strategy specifically refers to the strategy of splicing the character units in the initial word set with adjacent word units or character units; correspondingly, the word set to be processed specifically refers to the word set that does not contain character units after splicing the character units in the initial word set.
[0110] Based on this, in order to avoid the influence of the character units after word segmentation on the accuracy of determining illegal words, we can first select an initial word set containing character units from each word set, and then splice the character units in the initial word set according to a preset splicing strategy to obtain a to-be-processed word set that does not contain character units, which can be used for subsequent illegal word processing operations.
[0111] Furthermore, when the character units in the initial word set are spliced according to the preset splicing strategy, the splicing is actually performed selectively according to the frequency of the character units. In this embodiment, the specific implementation is as follows:
[0112] Determine the frequency corresponding to the character units contained in the initial word set and the adjacent character units associated with the character units; splice the character units and the adjacent character units according to the frequency, and obtain the to-be-processed word set according to the splicing result.
[0113] Specifically, the frequency corresponding to the word unit specifically refers to the frequency of occurrence of each character unit in the initial word set; correspondingly, the adjacent character unit specifically refers to other character units adjacent to the character unit in the initial word set, and the adjacent relationship can be forward adjacent or backward adjacent.
[0114] Based on this, after determining the initial word set containing the character unit, the frequency corresponding to the character unit and the adjacent character units associated with the character unit can be determined. Then, the character unit and the adjacent character units can be spliced according to the frequency to form a new word unit. The initial word set containing the new word unit can be used as the word set to be processed for subsequent illegal word determination and processing operations.
[0115] It should be noted that when performing splicing based on frequency, character units with a frequency greater than a threshold are actually selected for splicing with adjacent character units, and character units with a frequency less than the threshold are still regarded as a unit of word unit set, thereby avoiding the problem of missing a single character unit that may also be an illegal word.
[0116] In summary, by constructing new word units based on frequency and adjacent character units and updating the word units to be processed, we can not only avoid the influence of single character units on the accurate determination of illegal words, but also fully consider the possibility that character units may also be illegal, thereby ensuring the accurate coverage of illegal words.
[0117] Step S1266: perform iterative calculation on the set of words to be processed using a preset word replacement algorithm.
[0118] Specifically, the process of iteratively calculating the word set to be processed using the word replacement algorithm is similar to the iterative calculation process of the word set corresponding to each sentence unit mentioned above. For the convenience of description in this embodiment, the same description content in the above implementation mode can be referred to, and no further details will be given here.
[0119] Continuing with the above example, by analyzing word sets S1 and S2, it is determined that both have character units. At this time, the character units in each word set and the adjacent character units associated with them can be spliced according to the preset splicing strategy, and the word set to be processed corresponding to word set S1 {Player A, although, participated in, many, leagues} and the word set to be processed corresponding to word set S2 {But, he, kicks, the ball, really stinks} are obtained according to the splicing result; further, the BPE algorithm is used to iteratively calculate each word set to be processed, and the remaining word units of the iterative calculation results are compared with the standard word list, so as to determine that "really stinks" is not matched successfully, and the word unit "really stinks" is selected as the illegal word of the text to be processed, so as to be written into the corresponding illegal word library in the football field for subsequent use.
[0120] In summary, by adopting the method of character unit splicing to reduce the influence of character units on the determination of illegal words, the accuracy of illegal word determination can be effectively improved, thereby satisfying the downstream business to use illegal words according to needs.
[0121] In addition, after the illegal words are determined, in order to avoid repeated illegal words or illegal words that are normal word units, preset filtering rules can be used for re-screening to ensure that the illegal words written into the illegal word library are correct and can be applied to the desensitization scenario. In this embodiment, the specific implementation is as follows:
[0122] The illegal words are filtered according to preset filtering rules to obtain target illegal words and write them into a target illegal word library; when the text to be desensitized is received, the text to be desensitized is processed based on the target illegal word library to obtain the target text.
[0123] Specifically, the filtering rules specifically refer to the rules for normalizing and / or re-determining the illegal words, so as to eliminate repeated illegal words through the filtering rules, and / or avoid word unit qualitative errors; accordingly, the target illegal words specifically refer to the illegal words that can be written into the target illegal word library; accordingly, the text to be desensitized specifically refers to the text that needs to replace the illegal words contained therein, and the text can be an article to be published by the user, a communication content to be released, etc., and the target text is the text obtained after the desensitization process.
[0124] Based on this, after obtaining the illegal words in the text to be processed, the illegal words can be filtered according to the preset filtering rules to obtain the target illegal words and write them into the target illegal word library. When the text to be desensitized is received, the target illegal words in the target illegal word library can be directly reused to determine the illegal words in the text to be desensitized, and replaced with preset characters to obtain the target text.
[0125] For example, by processing the text to be processed, the illegal words include {P1, P2, P1, P3}, and then the repeated illegal words are removed according to the preset filtering rules, and the illegal words with inaccurate qualitative characteristics are removed. Finally, the target illegal words are determined to be P1 and P2, and they are written into the corresponding target illegal word library. Further, after receiving the communication content {zxcvbnp1} to be sent by the user, it can be desensitized based on the target illegal word library to obtain the target text {zxcvbn*}, and then sent.
[0126] In summary, by combining filtering rules to determine the target illegal words, the number of redundant illegal words can be reduced. At the same time, desensitization based on this can prevent illegal words from appearing in inappropriate scenarios, making it easier for downstream businesses to use them.
[0127] See also Figure 2 As shown in the schematic diagram, the text processing method provided by the present application can realize that when it is necessary to screen out illegal words, after obtaining the text to be processed, the text to be processed can be pre-processed to obtain the sentence units corresponding to the text to be processed and the word sets corresponding to each sentence unit, and then the sentence units with normal semantics are filtered out by using a preset word list, and the remaining sentence units are processed by a word replacement algorithm, so as to obtain the initial illegal words from the text to be processed, and finally the initial illegal words are deeply filtered by the rules, that is, the word units that can be written into the illegal word library can be obtained, so as to obtain the illegal words from the text to be processed, which can not only save the consumption of manpower and material resources, but also ensure the accuracy of illegal word filtering, realize the determination of illegal words in an unsupervised scenario, and facilitate the downstream to perform subsequent processing operations based on the illegal words obtained by screening.
[0128] The following combination Figure 3 , taking the application of the text processing method provided by the present application in the field of articles as an example, the text processing method is further explained. Figure 3 A processing flow chart of another text processing method provided by an embodiment of the present application is shown, which specifically includes the following steps:
[0129] Step S302, obtaining the text to be processed.
[0130] This embodiment uses the text to be processed "qwertyuiop, asdfghjkl; zxcvbnm." as an example to illustrate the text processing method, where each letter represents a Chinese character and is replaced by letters for the convenience of description. In actual applications, the text content contained in the text to be processed may be much larger than the text content illustrated in this embodiment. For the convenience of description, only part of the text content is described as the text to be processed.
[0131] Step S304, segmenting the text to be processed into sentences to obtain a plurality of sentence units.
[0132] The text to be processed is divided into sentences by identifying the punctuation marks contained in the text to be processed. That is, the text to be processed "qwertyuiop, asdfghjkl; zxcvbnm." contains punctuation marks {,;.}; the text to be processed is divided into sentences with each punctuation mark as a segmentation position, and the first sentence unit is {qwertyuiop}, the second sentence unit is {asdfghjkl}, and the third sentence unit is {zxcvbnm}.
[0133] Step S306: Use a preset word segmentation tool to perform word segmentation processing on each sentence unit to obtain a word set corresponding to each sentence unit.
[0134] The jieba word segmentation tool is used to segment the first sentence unit {qwertyuiop}, the second sentence unit {asdfghjkl} and the third sentence unit {zxcvbnm} respectively to obtain the word set corresponding to each sentence unit.
[0135] Among them, the word set S1 corresponding to the first sentence unit {qwertyuiop} is {qw, e, r, ty, ui, op};
[0136] The word set S2 corresponding to the second sentence unit {asdfghjkl} is {asd, f, g, hj, kl};
[0137] The word set S3 corresponding to the third sentence unit {zxcvbnm} is {zx, cv, bnm}.
[0138] Step S308: compare the word set corresponding to each sentence unit with a preset standard word list, and determine the illegal sentence unit according to the comparison result.
[0139] The word set S1{qw, e, r, ty, ui, op} corresponding to the first sentence unit is compared with the standard vocabulary, and it is determined that the word units e and r do not exist in the standard vocabulary, then the first sentence unit is determined as the first illegal sentence unit.
[0140] Similarly, the word set S2{asd, f, g, hj, kl} corresponding to the second sentence unit is compared with the standard vocabulary, and it is determined that the word units f and g do not exist in the standard vocabulary, then the second sentence unit is determined as the second illegal sentence unit.
[0141] Then, the word set S3 {zx, cv, bnm} corresponding to the third sentence unit is compared with the standard word list to determine that all the word units contained in the third sentence unit are in the word list, and then the sentence unit is filtered out. Only the first illegal sentence unit and the second illegal sentence unit are retained for subsequent determination of illegal words.
[0142] Step S310, using the BPE algorithm to iteratively calculate the word set corresponding to the illegal sentence unit, and determine the illegal words according to the iterative calculation result.
[0143] Determine the word set S1{qw, e, r, ty, ui, op} corresponding to the first illegal sentence unit, and the word set S2{asd, f, g, hj, kl} corresponding to the second illegal sentence unit; first determine the word units e and r containing single words in the word set S1, and determine the frequency of occurrence of e and r in the text to be processed, and determine that both are greater than the preset frequency threshold. At this time, e can be selected to be spliced with the adjacent word unit r to generate a new word unit er, and the word set S1{qw, e, r, ty, ui, op} is updated based on the new word unit er to obtain the word set S1-1{qw, er, ty, ui, op} corresponding to the first illegal sentence unit.
[0144] Similarly, determine the word units f and g that contain single words in the word set S2, and determine the frequency of occurrence of f and g in the text to be processed, and determine that both are greater than the preset frequency threshold. At this time, you can choose f and the adjacent word unit g for splicing to generate a new word unit fg, and update the word set S2{asd, f, g, hj, kl} based on the new word unit fg to obtain the word set S2-2{asd, fg, hj, kl} corresponding to the second illegal sentence unit.
[0145] Furthermore, the word sets corresponding to the two illegal sentence units are replaced in combination with the BPE algorithm, and the word units that can be replaced in the word set are all from the preset standard word list. In other words, the word units in the corresponding standard word list in the word set are selected for replacement, and the remaining word units are the initial illegal words.
[0146] Based on this, the BPE algorithm is used to replace the word units in the word set S1-1 {qw, er, ty, ui, op} corresponding to the first illegal sentence unit, and the remaining word units in the word set S1-1 are determined to be "er"; at the same time, the word units in the word set S2-2 {asd, fg, hj, kl} corresponding to the second illegal sentence unit are replaced, and the remaining word units in the word set S2-2 are determined to be "asd", then "er" and "asd" can be regarded as illegal words.
[0147] In addition, when the BPE algorithm is used to iteratively calculate the word set corresponding to the offending sentence unit, the offending word can also be determined by direct replacement. That is, each word unit in the word set is replaced according to the frequency of its appearance in the text to be processed, and the word with the highest frequency is replaced first; through iterative replacement, the last remaining word unit in the offending sentence unit is determined to be the offending word. Since the standard word list is not considered in the iterative replacement process, after the offending word is determined, the offending word can be compared with the standard word list, and then the word units in the offending word that appear in the standard word list are filtered out, and the remaining ones can be used as the target offending word.
[0148] For example, the word units in the word set S1-1 {qw, er, ty, ui, op} are directly replaced by the BPE algorithm, and the remaining word units are "qw" and "er"; similarly, the word units in the word set S2-2 {asd, fg, hj, kl} are replaced, and the remaining word units are "asd" and "fg"; at this time, "qw", "er", "asd" and "fg" are taken as initial illegal words; then the initial illegal words are compared with the standard word list one by one to determine that "qw" and "fg" both exist in the standard word list, then "qw" and "fg" are removed from the initial illegal words, and the remaining word units "er" and "asd" are the target illegal words, which can be used to expand the illegal word set.
[0149] Step S312: Filter the illegal words according to the preset rules to obtain target illegal words, and write the target illegal words into the illegal word set.
[0150] After obtaining the illegal words "er" and "asd" in the text to be processed, the illegal words "er" and "asd" can be filtered again according to the preset rules to select the target illegal word "er" and write it into the illegal words, thereby expanding the illegal word set.
[0151] In specific implementation, the preset rules may be semantic analysis of the offending words, or sensitive character / word analysis, or model detection, which is not limited in this embodiment.
[0152] Step S314: Process the text input by the user using the illegal word set to obtain the target text.
[0153] After receiving an article to be published uploaded by a user, in order to prevent the illegal words contained in the article from affecting the readers, the article can be processed using the illegal word set. If it is determined that the article contains the illegal word "er", the illegal word "er" in the article can be replaced with "**". According to the replacement result, the target article that does not contain the illegal word "er" can be obtained.
[0154] In summary, when it is necessary to update the set of illegal words, after obtaining the text to be processed, the text to be processed can be preprocessed to obtain the sentence units corresponding to the text to be processed and the word sets corresponding to each sentence unit, and then the sentence units with normal semantics are filtered out using the preset vocabulary, and the remaining sentence units are processed using the BPE algorithm, so as to obtain the initial illegal words from the text to be processed, and finally the rules are used to deeply filter the initial illegal words, that is, the word units that can be written into the illegal word set can be obtained, which can not only save manpower and material resources, but also ensure the update efficiency of the illegal word set, realize the expansion of illegal words in an unsupervised scenario, and better meet the use of downstream businesses.
[0155] Corresponding to the above method embodiment, the present application also provides a text processing device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of a text processing device provided by an embodiment of the present application. Figure 4 As shown, the device comprises:
[0156] An acquisition module 402 is configured to acquire a text to be processed;
[0157] The processing module 404 is configured to pre-process the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit;
[0158] The determination module 406 is configured to perform iterative calculation on the word set corresponding to each sentence unit using a preset word replacement algorithm, and determine the illegal words in the text to be processed according to the iterative calculation result.
[0159] In an optional embodiment, the processing module 404 is further configured to:
[0160] The text to be processed is segmented into sentences to obtain at least one sentence unit corresponding to the text to be processed; each sentence unit is segmented into words to obtain a word set corresponding to each sentence unit.
[0161] In an optional embodiment, the determining module 406 is further configured to:
[0162] The remaining word units in the word set corresponding to each sentence unit are determined according to the iteration result; and the remaining word units in the word set corresponding to each sentence unit are used as the illegal words.
[0163] In an optional embodiment, the text processing device further includes:
[0164] A matching module is configured to match the word set corresponding to each sentence unit with a preset standard word list; select a word set that does not match the standard word list as a target word set;
[0165] Wherein, the determining module 406 is further configured to:
[0166] The target word set is iteratively calculated using a preset word replacement algorithm.
[0167] In an optional embodiment, the determining module 406 is further configured to:
[0168] The remaining word units in the target word set are determined according to the iteration result; the remaining word units in the target word set are matched with the standard word list, and unmatched word units are selected as the offending words.
[0169] In an optional embodiment, the acquisition module 402 is further configured to:
[0170] Acquire the text to be processed corresponding to the target field;
[0171] Wherein, the determining module 406 is further configured to:
[0172] The illegal words are added to the illegal word library corresponding to the target domain.
[0173] In an optional embodiment, the text processing device further includes:
[0174] The splicing module is configured to select an initial word set containing character units from the word set corresponding to each sentence unit; splice the character units contained in the initial word set according to a preset splicing strategy to obtain a word set to be processed;
[0175] Wherein, the determining module 406 is further configured to:
[0176] The preset word replacement algorithm is used to iteratively calculate the word set to be processed.
[0177] In an optional embodiment, the splicing module is further configured as follows:
[0178] Determine the frequency corresponding to the character units contained in the initial word set and the adjacent character units associated with the character units; splice the character units and the adjacent character units according to the frequency, and obtain the to-be-processed word set according to the splicing result.
[0179] In an optional embodiment, the text processing device further includes:
[0180] The filtering module is configured to filter the illegal words according to preset filtering rules, obtain target illegal words and write them into a target illegal word library; when receiving the text to be desensitized, the text to be desensitized is processed based on the target illegal word library to obtain the target text.
[0181] The text processing device provided by the present application can realize that when it is necessary to filter out illegal words, after obtaining the text to be processed, the text to be processed can be pre-processed to obtain sentence units corresponding to the text to be processed and word sets corresponding to each sentence unit, and then a preset word replacement algorithm is used to iteratively calculate the word set corresponding to each sentence unit, so as to obtain illegal words from the text to be processed. This can not only save manpower and material resources, but also ensure the accuracy of illegal word filtering, and realize the determination of illegal words in an unsupervised scenario, so as to facilitate the downstream to perform subsequent processing operations based on the screened illegal words.
[0182] The above is a schematic scheme of a text processing device of the present embodiment. It should be noted that the technical scheme of the text processing device and the technical scheme of the above-mentioned text processing method belong to the same concept. For details not described in detail in the technical scheme of the text processing device, please refer to the description of the technical scheme of the above-mentioned text processing method. In addition, each component in the device embodiment should be understood as a functional module that must be established to implement each step of the program flow or each step of the method, and each functional module is not an actual functional division or separation definition. The device claim defined by such a group of functional modules should be understood as a functional module architecture that mainly implements the solution through the computer program recorded in the specification, and should not be understood as a physical device that mainly implements the solution through hardware.
[0183] Figure 5 The block diagram of a computing device 500 provided according to an embodiment of the present application is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.
[0184] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0185] In one embodiment of the present application, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.
[0186] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.
[0187] The processor 520 is used to execute the following computer executable instructions:
[0188] Get the text to be processed;
[0189] Preprocessing the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit;
[0190] The word set corresponding to each sentence unit is iteratively calculated using a preset word replacement algorithm, and the illegal words in the text to be processed are determined according to the iterative calculation results.
[0191] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above text processing method.
[0192] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which are used when executed by a processor to:
[0193] Get the text to be processed;
[0194] Preprocessing the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit;
[0195] The word set corresponding to each sentence unit is iteratively calculated using a preset word replacement algorithm, and the illegal words in the text to be processed are determined according to the iterative calculation results.
[0196] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above text processing method.
[0197] An embodiment of the present application further provides a chip storing a computer program, which implements the steps of the text processing method when executed by the chip.
[0198] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0199] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0200] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0201] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0202] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.
Claims
1. A text processing method, characterized in that: include: Get the text to be processed; Preprocessing the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit; The word set corresponding to each sentence unit is iteratively calculated using a preset word replacement algorithm, and the illegal words in the text to be processed are determined according to the iterative calculation results.
2. The text processing method according to claim 1, characterized in that: The preprocessing of the to-be-processed text to obtain at least one sentence unit and a word set corresponding to each sentence unit includes: Perform sentence processing on the text to be processed to obtain at least one sentence unit corresponding to the text to be processed; Each sentence unit is segmented to obtain the word set corresponding to each sentence unit.
3. The text processing method according to claim 1, characterized in that: Determining the illegal words in the text to be processed according to the iterative calculation result includes: Determine the remaining word units in the word set corresponding to each sentence unit according to the iteration result; The remaining word units in the word set corresponding to each sentence unit are regarded as the illegal words.
4. The text processing method according to claim 1, characterized in that: Before the step of iteratively calculating the word set corresponding to each sentence unit using a preset word replacement algorithm is performed, the method further includes: Match the word set corresponding to each sentence unit with a preset standard word list; Selecting a word set that does not match the standard word list as a target word set; The iterative calculation of the word set corresponding to each sentence unit by using a preset word replacement algorithm includes: The target word set is iteratively calculated using a preset word replacement algorithm.
5. The text processing method according to claim 4, characterized in that: The step of determining the illegal words in the text to be processed according to the iterative calculation result includes: Determining the remaining word units in the target word set according to the iteration result; The remaining word units in the target word set are matched with the standard word list, and unmatched word units are selected as the offending words.
6. The text processing method according to claim 1, characterized in that: The step of obtaining the text to be processed includes: Acquire the text to be processed corresponding to the target field; Wherein, after the step of determining the illegal words in the text to be processed according to the iterative calculation result is executed, the method further includes: The illegal words are added to the illegal word library corresponding to the target domain.
7. The text processing method according to claim 1, characterized in that: Before the step of iteratively calculating the word set corresponding to each sentence unit using a preset word replacement algorithm is performed, the method further includes: Filtering an initial word set containing character units from the word set corresponding to each sentence unit; splicing the character units contained in the initial word set according to a preset splicing strategy to obtain a word set to be processed; The iterative calculation of the word set corresponding to each sentence unit by using a preset word replacement algorithm includes: The preset word replacement algorithm is used to iteratively calculate the word set to be processed.
8. The text processing method according to claim 7, characterized in that: The step of splicing the character units contained in the initial word set according to a preset splicing strategy to obtain a word set to be processed includes: Determine the frequencies corresponding to the character units included in the initial word set, and the adjacent character units associated with the character units; The character unit and the adjacent character unit are spliced according to the frequency, and the to-be-processed word set is obtained according to the splicing result.
9. The text processing method according to any one of claims 1 to 8, characterized in that: After the step of determining the illegal words in the text to be processed according to the iterative calculation result is performed, the method further includes: Filter the illegal words according to the preset filtering rules, obtain the target illegal words and write them into the target illegal word library; When the text to be desensitized is received, the text to be desensitized is processed based on the target violation library to obtain the target text.
10. A text processing device, characterized in that: include: An acquisition module, configured to acquire text to be processed; A processing module is configured to pre-process the text to be processed to obtain at least one sentence unit and a word set corresponding to each sentence unit; The determination module is configured to perform iterative calculation on the word set corresponding to each sentence unit using a preset word replacement algorithm, and determine the illegal words in the text to be processed according to the iterative calculation result.
11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of any one of the methods of claims 1 to 9.
12. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the method described in any one of claims 1 to 9 are implemented.
