Method for extending sound near-sensitive words

Abstract

The application provides an extension method of homophonic sensitive words, comprising: combining two by two of pinyins in a legal pinyin table; obtaining an edit distance of each two-by-two combination result, and extracting a homophonic pinyin group according to the edit distance to construct a pinyin-homophonic pinyin table; replacing any character pinyin in a sensitive word in a sensitive word database based on the pinyin-homophonic pinyin table, and mapping the replaced any character pinyin into a character based on a pinyin-Hanzi table to construct a candidate homophonic word; and pre-judging the candidate homophonic word to realize the supplementary extension of the sensitive word database. By using the existing sensitive word library and homophonic word table, the homophonic character variants of the sensitive words that the black production may use are speculated, the characteristics such as the possibility of missing and the long time consumption of the whole link are solved in advance, and the effectiveness of the extracted keywords is improved.

Description

Technical Field

[0001] This invention relates to the field of vocabulary expansion technology, and in particular to a method for expanding words that are sensitive to similar sounds. Background Technology

[0002] In the context of online communities, due to the unique nature of the environment, a single message can often be received by a large number of people. This leads to the frequent occurrence of messages in the form of text or images used to redirect traffic to illegal platforms. These redirecting messages are generally considered spam, and their publishers are often considered to be involved in black market activities. Because these messages often contain similar keywords, these words are identified as sensitive words. By identifying whether the text contains these sensitive words, violations can be dealt with promptly, thus maintaining a healthy community atmosphere—that is, anti-spam content. However, those who post violating content often try to avoid these extracted sensitive words through experimental methods, reducing the number of times sensitive words are detected. Therefore, updating the sensitive word database is crucial for content anti-cheating. Currently, most platforms supplement their sensitive word databases manually.

[0003] While manually extracting keywords to supplement the sensitive word database can have some anti-cheating effect, this method cannot solve the problem proactively. This is because each sensitive word originates from the black market and is then manually extracted, a time-consuming process that cannot guarantee the identification of every violating text, resulting in numerous omissions. Even if keywords are successfully extracted, the rapid updates to black market tactics mean that newly added keywords may not be effective.

[0004] Therefore, this invention proposes an expansion method for sensitive words with similar pronunciations. Summary of the Invention

[0005] This invention provides an extended method for identifying phonetic sensitive words. By using an existing sensitive word database and a list of phonetic similar words, it infers phonetic variants of sensitive words that black market operators may use, thereby proactively addressing the high likelihood of omissions and time-consuming processes in the entire process, and thus improving the effectiveness of the extracted keywords.

[0006] This invention provides a method for expanding the scope of phonetic sensitive words, including:

[0007] Step 1: Combine the pinyin in the valid pinyin table in pairs;

[0008] Step 2: Obtain the edit distance of each pairwise combination result, and extract the near-phonetic pinyin groups based on the edit distance to construct a pinyin-near-phonetic pinyin table;

[0009] Step 3: Replace the pinyin of any character in the sensitive words in the sensitive word database based on the pinyin-phonetic similarity pinyin table, and map the replaced pinyin of any character into text based on the pinyin-Chinese character table to construct candidate phonetic similarity words;

[0010] Step 4: Pre-judgment of the candidate homophones to supplement and expand the sensitive word database.

[0011] In one possible implementation, the edit distance of each pairwise combination result is obtained, and near-phonetic pinyin groups are extracted based on the edit distance to construct a pinyin-near-phonetic pinyin table, including:

[0012] Calculate the edit distance between pairwise combinations;

[0013] Determine whether the edit distance is greater than a preset distance;

[0014] If so, determine that the first and second pinyin in the corresponding combination result are not similar in sound;

[0015] Otherwise, determine that the first and second pinyin in the corresponding combination result are similar in pronunciation;

[0016] Extract all phonetic similarity groups of pinyin and construct a pinyin-phonetic similarity table.

[0017] In one possible implementation, pre-judgment is performed on the candidate homophones to supplement and expand the sensitive word database, including:

[0018] Obtain the human judgment instruction for candidate homophones, and determine whether the judgment instruction is related to preset sensitive conditions;

[0019] When the judgment instruction is related to the preset sensitive conditions, the candidate sensitive words are automatically added to the sensitive word database;

[0020] When the judgment instruction is unrelated to the preset sensitive conditions, the candidate sensitive words are discarded.

[0021] In one possible implementation, the legal pinyin table and the pinyin-Chinese character table only include the pronunciation pinyin, excluding the pronunciation tones.

[0022] In one possible implementation, the pinyin in the valid pinyin table is combined in pairs, including:

[0023] The first category is based on the length of each pinyin syllable;

[0024] For each pinyin in the first category result, a second category is performed based on the type of the first letter of the pinyin;

[0025] Based on the first and second classification results, a set of pinyin with different pinyin lengths is obtained;

[0026] The pinyin in the set of pinyin are combined in pairs.

[0027] In one possible implementation, when the judgment instruction is related to a preset sensitive condition, the candidate sensitive words are automatically added to the sensitive word database, including:

[0028] The manual judgment instructions for candidate homophones are captured, and the judgment instructions are parsed to obtain several sub-judgment conditions;

[0029] Extract the current judgment differences for each sub-judgment condition;

[0030] Based on preset sensitive conditions, determine whether the current judgment difference is within the preset difference range. If so, determine that the corresponding sub-judgment condition is qualified.

[0031] When all the sub-judgment conditions are met, the judgment instruction is related to the preset sensitive conditions.

[0032] In one possible implementation, the process of pre-judging the candidate homophones to supplement and expand the sensitive word database also includes:

[0033] Tracing the sensitive word database, extracting initial historical sensitive words related to the candidate homophones, constructing initial derivative chains for different initial historical sensitive words, and optimizing the initial derivative chains based on the results of web sensitive word crawling to obtain historical derivative chains;

[0034] Determine whether there are letter characters in the initial sensitive words of each historical derivative chain, and regard the historical derivative chain with letter characters as the first type, and re-determine whether there are letter characters in the remaining historical sensitive words in the historical derivative chain where there are no letter characters in the initial sensitive words;

[0035] Based on the results of the second assessment, historical derivative chains that do not contain letter characters in the remaining sensitive words are extracted and regarded as the second category. At the same time, the remaining historical derivative chains are regarded as the third category.

[0036] Determine the initial historical pinyin corresponding to the initial historical sensitive word in the historical derivative chain and the modified historical pinyin after each derivative, and obtain historical pinyin tracing information;

[0037] By statistically analyzing the historical search time sets and historical search frequency sets of different historical sensitive words in the same historical derivative chain, a first curve graph of the same historical derivative chain is constructed.

[0038] Based on the historical pinyin tracing information, the total historical derivation time of the same historical derivation chain and the derivation time of each derivation stage are obtained, and a second curve is constructed.

[0039] Based on the curve analysis model, the first curve and the second curve are pre-analyzed. At the same time, combined with the classification type of the corresponding historical derivative chain, the first sensitivity contribution degree of different historical sensitive words in the corresponding historical derivative chain and the second sensitivity contribution degree of the corresponding historical derivative chain are obtained.

[0040] Extract first sensitive words from the same historical derivative chain whose first sensitive contribution level is greater than the first preset sensitive contribution level, and sort all first sensitive words in the same historical derivative chain according to the first sensitive contribution level;

[0041] When the second sensitivity contribution level is greater than the second preset sensitivity level, a first mapping relationship between the corresponding first sensitive word and the corresponding historical derivative chain is established based on the ranking result;

[0042] Otherwise, based on the sorting results, establish a second mapping relationship for the corresponding first sensitive word;

[0043] Based on the sensitive change capture model, all first and second mapping relationships are captured and analyzed to obtain sensitive change patterns. Based on these sensitive change patterns, an auxiliary change list of sensitive words is constructed.

[0044] Based on the sensitive word auxiliary change list, determine the word adjustment type for the candidate homophones;

[0045] Based on the vocabulary adjustment type, the corresponding homophones to be adjusted are determined.

[0046] In one possible implementation, the first sensitivity contribution level of different historical sensitive words in the same historical derivative chain and the second sensitivity contribution level of the corresponding historical derivative chain are obtained, including:

[0047] Calculate the degree of first sensitivity contribution B of different historical sensitive words in the same historical derivative chain according to the following formula. i (t,p);

[0048]

[0049] Where, sum{t i} represents the total search duration of the historical search time set of the i-th historical sensitive word in the same historical derivative chain; t i This represents the historical search time set of the corresponding i-th sensitive historical word; sum{p i} represents the total search frequency of the historical search frequency set of the i-th historical sensitive word in the same historical derivative chain; p i G represents the set of historical search frequencies for the corresponding i-th sensitive historical word; i T represents the search frequency of the i-th historical sensitive word; i This represents the derivative duration corresponding to the i-th historical sensitive word in the same historical derivative chain; This indicates the total duration of the historical derivation of the corresponding historical derivation chain; This represents the total search frequency of all historical sensitive words in the corresponding historical derivative chain;

[0050] Calculate the second sensitivity contribution level Y of the corresponding historical derivative chain according to the following formula;

[0051]

[0052] in, This indicates the classification type of the corresponding historical derivative chain. The value is 0.9 for the first category, 0.8 for the second, and 1 for the third. N represents the total number of historically sensitive words contained in the corresponding historical derivative chain. A i B represents the historical modification distance of the i-th historical sensitive word in the same historical derivative chain; i (t,p) represents the first sensitivity contribution level of the i-th historical sensitive word in the same historical derivative chain, based on the historical search time set t and the historical search frequency set p; This represents the total sensitivity contribution of all historical sensitive words in the same historical derivative chain, based on the historical search time set t and the historical search frequency set p. This represents the time-derived factor corresponding to the i-th historical sensitive word in the same historical derivative chain determined based on the second curve graph; This represents the sensitivity contribution factor corresponding to the i-th historical sensitive word in the same historical derivative chain determined based on the first curve graph;

[0053] First sensitive words are extracted based on the first and second sensitivity contribution levels.

[0054] In one possible implementation, the current judgment difference refers to the difference between the actual judgment result of the human on the same sub-judgment condition and the preset judgment standard of the sub-judgment condition.

[0055] In one possible implementation, the process of replacing the pinyin of any character in a sensitive word database based on the pinyin-phonetic similarity pinyin table includes:

[0056] Determine the first word extension subset for each second sensitive word in the sensitive word database, and construct the first extension sublist for each first word extension subset;

[0057] For each first extended sublist, complete overlap processing of sensitive words is performed, and several second extended subsets with the largest coverage are obtained. The second extended subsets include: a first extended subset consisting of only a single sensitive word and a second extended subset consisting of multiple sensitive words.

[0058] Determine the first number of the first extended subset and the second number of the second extended subset, and compare the first number with the second number;

[0059] If the first number is greater than or equal to the second number, the first replacement tag is set for the first extended subset according to the frequency of use of sensitive words in the first extended subset, and the pinyin of any character in the sensitive words in the corresponding first extended subset is replaced in sequence according to the tag replacement time attribute of the first replacement tag.

[0060] Otherwise, the central sensitive word in the second extended subset is determined first, and the extended meaning of each extended sensitive word based on the central sensitive word is determined, as well as the first pinyin modification position-time set of each extended sensitive word based on the central sensitive word and the second pinyin modification position-time set based on adjacent sensitive words;

[0061] Based on the extended meaning, the first pinyin modification position-time set, and the second pinyin modification position-time set, the degree of extension of the corresponding extended sensitive words is obtained;

[0062] According to the standard conversion rules, the extension degree of each extended sensitive word in the corresponding second extended subset is converted into the corresponding extended sequence, and the extended sequence of the corresponding second extended subset is obtained according to the order in which the extended sensitive words are generated.

[0063] Based on the consistency of the size of each sequence in the extended sequence, the extended sequence is divided into several sub-regions, and a second replacement label is set for the corresponding sub-region according to the mode sequence of each sub-region.

[0064] Based on the tag replacement time attribute of the second replacement tag, the pinyin of any character in the extended sensitive words of different sub-regions in the same second extended subset is replaced sequentially.

[0065] Compared with the prior art, the beneficial effects of this application are as follows:

[0066] By using existing sensitive word databases and homonym lists, we can infer homonymous variants of sensitive words that black market operators might use, thus proactively addressing the high likelihood of omissions and time-consuming processes in the entire process, thereby improving the effectiveness of the extracted keywords.

[0067] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0068] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0069] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0070] Figure 1 This is a flowchart of a method for expanding the sensitivity of similar-sounding words according to an embodiment of the present invention;

[0071] Figure 2 This is a specific implementation diagram of a method for expanding the sensitive words with similar sounds according to an embodiment of the present invention. Detailed Implementation

[0072] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0073] Example 1:

[0074] This invention provides a method for expanding the scope of phonetic sensitive words, such as... Figure 1 As shown, it includes:

[0075] Step 1: Combine the pinyin in the valid pinyin table in pairs;

[0076] Step 2: Obtain the edit distance of each pairwise combination result, and extract the near-phonetic pinyin groups based on the edit distance to construct a pinyin-near-phonetic pinyin table;

[0077] Step 3: Replace the pinyin of any character in the sensitive words in the sensitive word database based on the pinyin-phonetic similarity pinyin table, and map the replaced pinyin of any character into text based on the pinyin-Chinese character table to construct candidate phonetic similarity words;

[0078] Step 4: Pre-judgment of the candidate homophones to supplement and expand the sensitive word database.

[0079] Preferably, the legal pinyin table and the pinyin-Chinese character table only include pronunciation pinyin and do not include pronunciation tones.

[0080] In this embodiment, the edit distance can be calculated by the Levenshtein method. For example, the pairwise combination result is: pinyin "li" and pinyin "ni". At this time, the edit distance between pinyin "li" and "ni" is 1, because it only takes one step to replace "l" with "n".

[0081] In this embodiment, the legal pinyin table can be composed of existing Chinese pinyin, and the pinyin-similar pinyin table, for example, is composed of pairwise combinations of pinyin similar to the above "li" and "ni", representing the mapping relationship between a pinyin and its similar pinyin. The pinyin-Chinese character table can also be composed of existing pinyin and the Chinese characters corresponding to the pinyin, representing the mapping relationship between a pinyin and its homophonic Chinese characters corresponding to its pronunciation.

[0082] In this embodiment, the sensitive word database is pre-constructed.

[0083] In this embodiment, the candidate similar pinyin words refer to the words obtained after pinyin similar replacement.

[0084] In this embodiment, the pre-judgment refers to manual judgment on whether the candidate similar pinyin words can be retained. If they can be retained, they are supplemented and extended to the corresponding sensitive word database. If they cannot be retained, they are directly eliminated. In this embodiment, based on the pinyin-similar pinyin table, any Chinese character pinyin in the sensitive words in the sensitive word database is replaced. For example: for the pinyin "zhihusou", the "zhi" in it can be replaced with "zi", and the homophone table is used to map the pinyin after the similar pinyin replacement into Chinese characters, that is, "zihusou" can be mapped into "字乎搜", and the specific process is shown in Figure 2 as shown.

[0085] In this embodiment, by using the method of defining similar pinyin with the Levenshtein distance to generate similar Chinese characters, and then expanding the sensitive word library, some possible variants of sensitive words are generated in advance. After manual review, the unreadable sensitive words are eliminated, and the readable variants are added to the sensitive word library to achieve the effect of identifying spam texts.

[0086] At the same time, compared with some solutions using complex language recognition models, such as Bert, the solution of adding sensitive words has the characteristics of fast iteration speed and short cycle.

[0087] The beneficial effects of the above technical solutions are: by using the existing sensitive word library and the similar pinyin word table, inferring the similar Chinese character variants of sensitive words that black production may use, to pre-solve the problems of high possibility of missed detection and long time consumption in the whole process, and thus improve the effectiveness of the extracted keywords.

[0088] Example 2:

[0089] Based on Example 1, the edit distance of each pairwise combination result is obtained, and near-phonetic pinyin groups are extracted according to the edit distance to construct a pinyin-near-phonetic pinyin table, including:

[0090] Calculate the edit distance between pairwise combinations;

[0091] Determine whether the edit distance is greater than a preset distance;

[0092] If so, determine that the first and second pinyin in the corresponding combination result are not similar in sound;

[0093] Otherwise, determine that the first and second pinyin in the corresponding combination result are similar in pronunciation;

[0094] Extract all phonetic similarity groups of pinyin and construct a pinyin-phonetic similarity table.

[0095] In this embodiment, the preset distance can be set to 2.

[0096] The beneficial effects of the above technical solution are: by comparing distances, it is possible to effectively determine which characters are similar in sound and which are not, providing a basis for subsequent speculation on variants of similar-sounding characters.

[0097] Example 3:

[0098] Based on Example 1, the candidate homophones are pre-judged to supplement and expand the sensitive word database, including:

[0099] Obtain the human judgment instruction for candidate homophones, and determine whether the judgment instruction is related to preset sensitive conditions;

[0100] When the judgment instruction is related to the preset sensitive conditions, the candidate sensitive words are automatically added to the sensitive word database;

[0101] When the judgment instruction is unrelated to the preset sensitive conditions, the candidate sensitive words are discarded.

[0102] In this embodiment, for example, if there is a sensitive word A1, the corresponding judgment instruction is that a certain character in the sensitive word A1 can be used as a sensitive word, and at this time, this sensitive word also meets the preset sensitive conditions, so it can be regarded as relevant. The criteria for timely expansion of sensitive words are also considered, such as whether they are easy to understand, whether their meaning is obvious, and whether they are ambiguous.

[0103] The beneficial effect of the above technical solution is that by analyzing the manual judgment instructions, it can be determined whether to discard candidate sensitive words, thus ensuring the rationality of the sensitive word database.

[0104] Example 4:

[0105] Based on Example 1, the pinyin in the legal pinyin table is combined in pairs, including:

[0106] The first category is based on the length of each pinyin syllable;

[0107] For each pinyin in the first category result, a second category is performed based on the type of the first letter of the pinyin;

[0108] Based on the first and second classification results, a set of pinyin with different pinyin lengths is obtained;

[0109] The pinyin in the set of pinyin are combined in pairs.

[0110] In this embodiment, for example, the pinyin is first classified according to its length: 2 lengths, 3 lengths, etc., and then classified again according to the priority order of the 26 letters in the 2 lengths to obtain a pinyin set. The pinyin sets are then combined in pairs, and the pinyin set refers to the set constructed from the same first letter under different lengths.

[0111] The beneficial effects of the above technical solution are: by classifying pinyin by length and by initial letter, the pinyin in the pinyin table can be effectively combined in pairs, improving the combination efficiency and thus improving the efficiency of obtaining the editing distance in the future.

[0112] Example 5:

[0113] Based on Embodiment 3, when the judgment instruction is related to preset sensitive conditions, the candidate sensitive words are automatically added to the sensitive word database, including:

[0114] The manual judgment instructions for candidate homophones are captured, and the judgment instructions are parsed to obtain several sub-judgment conditions;

[0115] Extract the current judgment differences for each sub-judgment condition;

[0116] Based on preset sensitive conditions, determine whether the current judgment difference is within the preset difference range. If so, determine that the corresponding sub-judgment condition is qualified.

[0117] When all the sub-judgment conditions are met, the judgment instruction is related to the preset sensitive conditions.

[0118] Preferably, the current judgment difference refers to the difference between the actual judgment result of the human judge on the same sub-judgment condition and the preset judgment standard of the sub-judgment condition.

[0119] In this embodiment, the sub-judgment condition can be a judgment condition such as whether a certain character in the sensitive words is a sensitive character, and the corresponding preset judgment standard is that if the sensitivity of the sensitive character reaches a certain value, it can be regarded as meeting the corresponding preset judgment standard.

[0120] The beneficial effects of the above technical solution are: by determining the difference between the sub-judgment conditions and the preset evaluation criteria, the candidate sensitive words can be automatically supplemented and expanded, thereby improving the accuracy of subsequent inference of homophonous variants.

[0121] Example 6:

[0122] Based on Example 1, the process of pre-judging the candidate homophones to supplement and expand the sensitive word database also includes:

[0123] Tracing the sensitive word database, extracting initial historical sensitive words related to the candidate homophones, constructing initial derivative chains for different initial historical sensitive words, and optimizing the initial derivative chains based on the results of web sensitive word crawling to obtain historical derivative chains;

[0124] Determine whether there are letter characters in the initial sensitive words of each historical derivative chain, and regard the historical derivative chain with letter characters as the first type, and re-determine whether there are letter characters in the remaining historical sensitive words in the historical derivative chain where there are no letter characters in the initial sensitive words;

[0125] Based on the results of the second assessment, historical derivative chains that do not contain letter characters in the remaining sensitive words are extracted and regarded as the second category. At the same time, the remaining historical derivative chains are regarded as the third category.

[0126] Determine the initial historical pinyin corresponding to the initial historical sensitive word in the historical derivative chain and the modified historical pinyin after each derivative, and obtain historical pinyin tracing information;

[0127] By statistically analyzing the historical search time sets and historical search frequency sets of different historical sensitive words in the same historical derivative chain, a first curve graph of the same historical derivative chain is constructed.

[0128] Based on the historical pinyin tracing information, the total historical derivation time of the same historical derivation chain and the derivation time of each derivation stage are obtained, and a second curve is constructed.

[0129] Based on the curve analysis model, the first curve and the second curve are pre-analyzed. At the same time, combined with the classification type of the corresponding historical derivative chain, the first sensitivity contribution degree of different historical sensitive words in the corresponding historical derivative chain and the second sensitivity contribution degree of the corresponding historical derivative chain are obtained.

[0130] Extract first sensitive words from the same historical derivative chain whose first sensitive contribution level is greater than the first preset sensitive contribution level, and sort all first sensitive words in the same historical derivative chain according to the first sensitive contribution level;

[0131] When the second sensitivity contribution level is greater than the second preset sensitivity level, a first mapping relationship between the corresponding first sensitive word and the corresponding historical derivative chain is established based on the ranking result;

[0132] Otherwise, based on the sorting results, establish a second mapping relationship for the corresponding first sensitive word;

[0133] Based on the sensitive change capture model, all first and second mapping relationships are captured and analyzed to obtain sensitive change patterns. Based on these sensitive change patterns, an auxiliary change list of sensitive words is constructed.

[0134] Based on the sensitive word auxiliary change list, determine the word adjustment type for the candidate homophones;

[0135] Based on the vocabulary adjustment type, the corresponding homophones to be adjusted are determined.

[0136] In this embodiment, constructing an auxiliary change list of sensitive words is intended to further supplement the sensitive word database, ensure the rationality of the supplementation, and greatly enrich the database's data foundation.

[0137] In this embodiment, for example, if there are candidate homophones 1, 2, and 3, then the initial historical sensitive words related to the candidate homophones 1, 2, and 3 are obtained from the sensitive word database, that is, the very first sensitive word is found. Based on the sensitive word database itself, the initial derivative chain of the initial historical sensitive word is obtained.

[0138] In this embodiment, due to the development of network technology, there may be some character replacements or other alternative names used to replace the original sensitive words in the process of identifying homophones. Therefore, by combining with the results of crawling sensitive words on the network, the historical derivative chain can be effectively obtained.

[0139] In this embodiment, since the initial derivative chain may be described in words, but after optimization, there may be cases where letters appear. Therefore, the type of the corresponding historical derivative chain can be determined by judging the letter characters of the initial sensitive words, the remaining sensitive words, etc.

[0140] In this embodiment, the determination of letter characters can provide a basis for identifying the change pattern of sensitive words. For example, the initial derivative chain is 00-01-02-1, and the optimized historical derivative chain is 00-01-001-02-002-1. The optimization can be to add sensitive words to the initial derivative chain.

[0141] In this embodiment, the initial historical pinyin refers to the initial sensitive word, and the modified historical pinyin after each derivation. For example, from 00 to 01, the modified historical pinyin of 01 for 00 is obtained, thus obtaining the historical pinyin tracing information, that is, the pinyin modification information of 00 to 01, 01 to 001, 001 to 002, and 002 to 1.

[0142] In this embodiment, the historical search time set refers to all search time points from the time the sensitive word was derived until the present, which constitutes the historical search time set, and all search counts from the time the sensitive word was derived until the present constitute the historical search frequency set.

[0143] In this embodiment, the first curve is composed of the search time point and the number of searches at each search time point, and the second curve is composed of the total duration and the segment duration. The total duration of historical derivation refers to the derivation duration from 00 to 1 in the historical derivation chain: 00-01-001-02-002-1, and the derivation duration of each segment from 00-01, 01-001, etc., is regarded as the derivation segment duration.

[0144] In this embodiment, the graph is pre-analyzed, mainly to determine the sensitivity contribution of sensitive words and their derivative chains.

[0145] In this embodiment, the first preset sensitivity contribution level and the second preset sensitivity level are both preset. If the second sensitivity contribution level is greater than the second preset sensitivity level, then the first sensitive words are sorted according to the first sensitivity contribution level, and a mapping relationship with the historical derivative chain is constructed according to the partial order result. Otherwise, only the mapping relationship of the first sensitive words is constructed, mainly to exclude the non-sensitive contribution of sensitive words.

[0146] In this embodiment, the sensitive change capture model is trained on samples based on various mapping relationships and corresponding rules. Therefore, sensitive change rules can be obtained by training the first and second mapping relationships, and then a sensitive word auxiliary change list can be obtained. The sensitive word auxiliary change list contains various different sensitive change bodies (changes in pinyin, letters, characters, etc.). In this way, the auxiliary class type that can be adjusted for candidate sensitive words can be determined. For example, the candidate homophone can be adjusted to be represented by a certain character. Finally, the sensitive word database is enriched by comprehensively judging the candidate homophone and the adjusted homophone.

[0147] The beneficial effects of the above technical solution are as follows: by constructing historical derivative chains and analyzing different curves, different mapping relationships can be effectively obtained. Then, through model analysis, adjustment types can be obtained, and then adjusted near-phonetic words can be obtained. Furthermore, this improves the candidate near-phonetic words obtained by simply changing the pinyin, which can further enrich the sensitive word database. In other words, it provides a foundation for solving the characteristics of the whole process, such as the high possibility of omission and long time consumption, and also indirectly improves the effectiveness of the extracted keywords.

[0148] Example 7:

[0149] Based on Example 6, the first sensitivity contribution degree of different historical sensitive words in the same historical derivative chain and the second sensitivity contribution degree of the corresponding historical derivative chain are obtained, including:

[0150] Calculate the degree of first sensitivity contribution B of different historical sensitive words in the same historical derivative chain according to the following formula. i (t,p);

[0151]

[0152] Where, sum{t i} represents the total search duration of the historical search time set of the i-th historical sensitive word in the same historical derivative chain; t i This represents the historical search time set of the corresponding i-th sensitive historical word; sum{p i} represents the total search frequency of the historical search frequency set of the i-th historical sensitive word in the same historical derivative chain; p i G represents the set of historical search frequencies for the corresponding i-th sensitive historical word; i T represents the search frequency of the i-th historical sensitive word; i This represents the derivative duration corresponding to the i-th historical sensitive word in the same historical derivative chain; This indicates the total duration of the historical derivation of the corresponding historical derivation chain; This represents the total search frequency of all historical sensitive words in the corresponding historical derivative chain;

[0153] Calculate the second sensitivity contribution level Y of the corresponding historical derivative chain according to the following formula;

[0154]

[0155] in, This indicates the classification type of the corresponding historical derivative chain. The value is 0.9 for the first category, 0.8 for the second, and 1 for the third. N represents the total number of historically sensitive words contained in the corresponding historical derivative chain. A i B represents the historical modification distance of the i-th historical sensitive word in the same historical derivative chain; i (t,p) represents the first sensitivity contribution level of the i-th historical sensitive word in the same historical derivative chain, based on the historical search time set t and the historical search frequency set p; This represents the total sensitivity contribution of all historical sensitive words in the same historical derivative chain, based on the historical search time set t and the historical search frequency set p. This represents the time-derived factor corresponding to the i-th historical sensitive word in the same historical derivative chain determined based on the second curve graph; This represents the sensitivity contribution factor corresponding to the i-th historical sensitive word in the same historical derivative chain determined based on the first curve graph;

[0156] First sensitive words are extracted based on the first and second sensitivity contribution levels.

[0157] In this embodiment, the time-derived factor and the sensitive contribution factor can be the relevant time proportion and the sensitive contribution proportion, and the value range is (0, 1).

[0158] In this embodiment, the historical modification distance should be 0, 1, or 2, mainly considering the case where the corresponding editing distance meets the preset distance.

[0159] The beneficial effects of the above technical solution are: by calculating the first sensitive contribution degree in the same historical derivative chain and the second sensitive contribution degree in the corresponding derivative chain according to the above formula, the subsequent sensitive words can be effectively extracted, the extraction efficiency can be improved, and the efficiency of establishing mapping relationships can also be improved, ensuring the reliability of subsequent inferred homophonic variants.

[0160] Example 8:

[0161] Based on Example 1, the process of replacing the pinyin of any character in a sensitive word database using the pinyin-phonetic similarity pinyin table includes:

[0162] Determine the first word extension subset for each second sensitive word in the sensitive word database, and construct the first extension sublist for each first word extension subset;

[0163] For each first extended sublist, complete overlap processing of sensitive words is performed, and several second extended subsets with the largest coverage are obtained. The second extended subsets include: a first extended subset consisting of only a single sensitive word and a second extended subset consisting of multiple sensitive words.

[0164] Determine the first number of the first extended subset and the second number of the second extended subset, and compare the first number with the second number;

[0165] If the first number is greater than or equal to the second number, the first replacement tag is set for the first extended subset according to the frequency of use of sensitive words in the first extended subset, and the pinyin of any character in the sensitive words in the corresponding first extended subset is replaced in sequence according to the tag replacement time attribute of the first replacement tag.

[0166] Otherwise, the central sensitive word in the second extended subset is determined first, and the extended meaning of each extended sensitive word based on the central sensitive word is determined, as well as the first pinyin modification position-time set of each extended sensitive word based on the central sensitive word and the second pinyin modification position-time set based on adjacent sensitive words;

[0167] Based on the extended meaning, the first pinyin modification position-time set, and the second pinyin modification position-time set, the degree of extension of the corresponding extended sensitive words is obtained;

[0168] According to the standard conversion rules, the extension degree of each extended sensitive word in the corresponding second extended subset is converted into the corresponding extended sequence, and the extended sequence of the corresponding second extended subset is obtained according to the order in which the extended sensitive words are generated.

[0169] Based on the consistency of the size of each sequence in the extended sequence, the extended sequence is divided into several sub-regions, and a second replacement label is set for the corresponding sub-region according to the mode sequence of each sub-region.

[0170] Based on the tag replacement time attribute of the second replacement tag, the pinyin of any character in the extended sensitive words of different sub-regions in the same second extended subset is replaced sequentially.

[0171] In this embodiment, the sensitive word database contains second sensitive words: 1, 2, 3, and 4. At this time, the corresponding first word extension subsets are constructed from the second sensitive words 1, 2, 3, and 4 respectively. For example, the first word extension subset corresponding to the second sensitive word 1 is 1 and 2, the first word extension subset corresponding to the second sensitive word 2 is 1, 2, and 3, the first word extension subset corresponding to the second sensitive word 3 is 2 and 3, and the first word extension subset corresponding to the second sensitive word 4 is 4. After performing complete overlap processing, the second extension subsets obtained are 4 and 1, 2, and 3. The number of each subset is one. At this time, either subset can be randomly selected for priority processing.

[0172] In this embodiment, the usage frequency of sensitive words refers to the search frequency, specifically the search frequency of sensitive words retrieved from the sensitive word database. A higher usage frequency indicates a greater likelihood of needing replacement, meaning the tags prioritized for replacement appear earlier. For example, if there are two first extended subsets, and the usage frequency of sensitive words in the first subset is higher than that in the second subset, then the sensitive words in the first subset can be replaced first. This process continues until all sensitive words in the first subset are replaced, before moving on to the second subset. By setting a priority replacement order, both the effectiveness of the replacement process and the efficiency of replacement are ensured.

[0173] In this embodiment, the central sensitive word in the second extended subset can be, for example, extracted during the complete overlap processing. For instance, in the first word extended subsets 1 and 2, 1, 2, 3, and 2, 3, the corresponding central sensitive word could be 2. That is, the sensitive word with the largest overlap is considered the central sensitive word. If there are multiple sensitive words with the same largest overlap, one is randomly selected as the central sensitive word.

[0174] In this embodiment, extended meaning refers to the network meaning of the extended sensitive word. The pinyin modification position-time set refers to a set consisting of the time and the letter position of the modified letter, and then the degree of extension is obtained. The more modification time points, the more modification positions, and the more sensitive the extended meaning, the deeper the corresponding degree of extension. That is, based on the standard conversion rules, the degree of extension is converted into a larger value of the extension sequence, which is generally (0, 1).

[0175] In this embodiment, for example, the extended sequence in the second extended subset is: 0 0.3 0.2 0.1 0 0 0 1 10.8. This sequence is divided into segments based on size consistency. For example, the sub-region 0 0.3 0.2 0.1 0 0 0 has a corresponding mode sequence of 0, and the sub-region 1 1 0.8 has a corresponding mode sequence of 1. Then, the label replacement time can be set according to the size of the corresponding mode sequence to determine the order.

[0176] The beneficial effects of the above technical solution are as follows: by determining the extended sublist and performing complete overlap processing, the first extended subset and the second extended subset can be effectively obtained. By setting replacement tags and determining the tag replacement time attribute, the order of pinyin replacement can be effectively performed. In the process of replacing pinyin in the second extended subset, it is necessary to obtain the extension degree of the extended sensitive words. This is used to determine the sequence and divide the second extended subset into regions. This ensures that replacement characters with a high degree of change are processed first, ensuring the efficiency of variants and improving the reliability of subsequent inference of homophone variants.

[0177] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for expanding the list of phonetically similar sensitive words, characterized in that, include: Step 1: Combine the pinyin in the valid pinyin table in pairs; Step 2: Obtain the edit distance of each pairwise combination result, and extract the near-phonetic pinyin groups based on the edit distance to construct a pinyin-near-phonetic pinyin table; Step 3: Replace the pinyin of any character in the sensitive words in the sensitive word database based on the pinyin-phonetic similarity pinyin table, and map the replaced pinyin of any character into text based on the pinyin-Chinese character table to construct candidate phonetic similarity words; Step 4: Pre-judgment of the candidate homophones to supplement and expand the sensitive word database; The process of pre-judging the candidate homophones to supplement and expand the sensitive word database also includes: Tracing the sensitive word database, initial historical sensitive words related to the candidate homophones are extracted, initial derivative chains of different initial historical sensitive words are constructed, and the initial derivative chains are optimized in combination with the results of web sensitive word crawling to obtain historical derivative chains; Determine whether there are letter characters in the initial sensitive words of each historical derivative chain, and regard the historical derivative chain with letter characters as the first type, and re-determine whether there are letter characters in the remaining historical sensitive words in the historical derivative chain where there are no letter characters in the initial sensitive words; Based on the results of the second assessment, historical derivative chains that do not contain letter characters in the remaining sensitive words are extracted and regarded as the second category. At the same time, the remaining historical derivative chains are regarded as the third category. Determine the initial historical pinyin corresponding to the initial historical sensitive word in the historical derivative chain and the modified historical pinyin after each derivative, and obtain historical pinyin tracing information; By statistically analyzing the historical search time sets and historical search frequency sets of different historical sensitive words in the same historical derivative chain, a first curve graph of the same historical derivative chain is constructed. Based on the historical pinyin tracing information, the total historical derivation time of the same historical derivation chain and the derivation time of each derivation stage are obtained, and a second curve is constructed. Based on the curve analysis model, the first curve and the second curve are pre-analyzed. At the same time, combined with the classification type of the corresponding historical derivative chain, the first sensitivity contribution degree of different historical sensitive words in the corresponding historical derivative chain and the second sensitivity contribution degree of the corresponding historical derivative chain are obtained. Extract first sensitive words from the same historical derivative chain whose first sensitive contribution level is greater than the first preset sensitive contribution level, and sort all first sensitive words in the same historical derivative chain according to the first sensitive contribution level; When the second sensitivity contribution level is greater than the second preset sensitivity level, a first mapping relationship between the corresponding first sensitive word and the corresponding historical derivative chain is established based on the ranking result; Otherwise, based on the sorting results, establish a second mapping relationship for the corresponding first sensitive word; Based on the sensitive change capture model, all first and second mapping relationships are captured and analyzed to obtain sensitive change patterns. Based on these sensitive change patterns, an auxiliary change list of sensitive words is constructed. Based on the sensitive word auxiliary change list, determine the word adjustment type for the candidate homophones; Based on the aforementioned vocabulary adjustment type, determine the corresponding homophones to be adjusted; This includes obtaining the first sensitivity contribution level of different historical sensitive words in the same historical derivative chain and the second sensitivity contribution level of the corresponding historical derivative chain, including: The following formula is used to calculate the degree of primary sensitivity contribution of different historically sensitive words in the same historical derivative chain. ; in, This represents the total search duration of the historical search time set for the i-th historical sensitive word in the same historical derivative chain; This represents the set of historical search times for the corresponding i-th sensitive historical keyword; This represents the total search frequency of the historical search frequency set of the i-th historical sensitive word in the same historical derivative chain; This represents the set of historical search frequencies for the corresponding i-th sensitive historical word; This represents the search frequency of the i-th historical sensitive word. This represents the derivative duration corresponding to the i-th historical sensitive word in the same historical derivative chain; This indicates the total duration of the historical derivation of the corresponding historical derivation chain; This represents the total search frequency of all historical sensitive words in the corresponding historical derivative chain; Calculate the second sensitivity contribution level Y of the corresponding historical derivative chain according to the following formula; in, This indicates the classification type of the corresponding historical derivative chain. When it is the first type, the value is 0.9; when it is the second type, the value is 0.8; and when it is the third type, the value is 1. N represents the total number of historically sensitive words contained in the corresponding historical derivative chain. This represents the historical modification distance of the i-th historical sensitive word in the same historical derivative chain; This indicates the first sensitivity contribution level of the i-th historical sensitive word in the same historical derivative chain, based on the historical search time set t and the historical search frequency set p. This represents the total sensitivity contribution of all historical sensitive words in the same historical derivative chain, based on the historical search time set t and the historical search frequency set p. This represents the time-derived factor corresponding to the i-th historical sensitive word in the same historical derivative chain determined based on the second curve graph; This represents the sensitivity contribution factor corresponding to the i-th historical sensitive word in the same historical derivative chain determined based on the first curve graph; First sensitive words are extracted based on the first and second sensitivity contribution levels.

2. The method for expanding the sensitive words with similar sounds as described in claim 1, characterized in that, Obtain the edit distance for each pairwise combination result, and extract the near-phonetic pinyin groups based on the edit distance to construct a pinyin-near-phonetic pinyin table, including: Calculate the edit distance between pairwise combinations; Determine whether the edit distance is greater than a preset distance; If so, determine that the first and second pinyin in the corresponding combination result are not similar in sound; Otherwise, determine that the first and second pinyin in the corresponding combination result are similar in pronunciation; Extract all phonetic similarity groups of pinyin and construct a pinyin-phonetic similarity table.

3. The method for expanding the sensitive words with similar sounds as described in claim 1, characterized in that, Pre-judgment of the candidate homophones is performed to supplement and expand the sensitive word database, including: Obtain the human judgment instruction for candidate homophones, and determine whether the judgment instruction is related to preset sensitive conditions; When the judgment instruction is related to the preset sensitive conditions, the candidate sensitive words will be automatically added to the sensitive word database; When the judgment instruction is unrelated to the preset sensitive conditions, the candidate sensitive words are discarded.

4. The method for expanding the sensitive words with similar sounds as described in claim 1, characterized in that, The legal pinyin table and pinyin-Chinese character table only include the pronunciation pinyin, not the pronunciation tones.

5. The method for expanding the sensitive words with similar sounds as described in claim 1, characterized in that, For each pair of pinyin in the valid pinyin table, the combinations include: The first category is based on the length of each pinyin syllable; For each pinyin in the first category result, a second category is performed based on the type of the first letter of the pinyin; Based on the first and second classification results, a set of pinyin with different pinyin lengths is obtained; The pinyin in the set of pinyin are combined in pairs.

6. The method for expanding the sensitive words with similar sounds as described in claim 3, characterized in that, When the judgment instruction is related to preset sensitive conditions, the candidate sensitive words are automatically added to the sensitive word database, including: The manual judgment instructions for candidate homophones are captured, and the judgment instructions are parsed to obtain several sub-judgment conditions; Extract the current judgment differences for each sub-judgment condition; Based on preset sensitive conditions, determine whether the current judgment difference is within the preset difference range. If so, determine that the corresponding sub-judgment condition is qualified. When all the sub-judgment conditions are met, the judgment instruction is related to the preset sensitive conditions.

7. The method for expanding the sensitive words with similar sounds as described in claim 6, characterized in that, The difference in current judgment refers to the difference between the actual judgment result of the human judge on the same sub-judgment condition and the preset judgment standard of the sub-judgment condition.

8. The method for expanding the sensitive words with similar sounds as described in claim 1, characterized in that, The process of replacing the pinyin of any character in a sensitive word in the sensitive word database based on the aforementioned pinyin-phonetic similarity pinyin table includes: Determine the first word extension subset for each second sensitive word in the sensitive word database, and construct the first extension sublist for each first word extension subset; For each first extended sublist, complete overlap processing of sensitive words is performed, and several second extended subsets with the largest coverage are obtained. The second extended subsets include: a first extended subset consisting of only a single sensitive word and a second extended subset consisting of multiple sensitive words. Determine the first number of the first extended subset and the second number of the second extended subset, and compare the first number with the second number; If the first number is greater than or equal to the second number, the first replacement tag is set for the first extended subset according to the frequency of use of sensitive words in the first extended subset, and the pinyin of any character in the sensitive words in the corresponding first extended subset is replaced in sequence according to the tag replacement time attribute of the first replacement tag. Otherwise, the central sensitive word in the second extended subset is determined first, and the extended meaning of each extended sensitive word based on the central sensitive word is determined, as well as the first pinyin modification position-time set of each extended sensitive word based on the central sensitive word and the second pinyin modification position-time set based on adjacent sensitive words; Based on the extended meaning, the first pinyin modification position-time set, and the second pinyin modification position-time set, the degree of extension of the corresponding extended sensitive words is obtained; According to the standard conversion rules, the extension degree of each extended sensitive word in the corresponding second extended subset is converted into the corresponding extended sequence, and the extended sequence of the corresponding second extended subset is obtained according to the order in which the extended sensitive words are generated. Based on the consistency of the size of each sequence in the extended sequence, the extended sequence is divided into several sub-regions, and a second replacement label is set for the corresponding sub-region according to the mode sequence of each sub-region. Based on the tag replacement time attribute of the second replacement tag, the pinyin of any character in the extended sensitive words of different sub-regions in the same second extended subset is replaced sequentially.

Images