Text information processing method, device, electronic device and medium

Abstract

The present disclosure provides a method, device, electronic device, and medium for processing text information, which relates to the field of artificial intelligence technology, specifically the field of natural language processing. The specific implementation scheme is: obtaining target text information to be processed; identifying target escaped words in the target text information based on a predetermined association relationship between candidate real words and candidate escaped words, and modifying the target escaped words to restore the target escaped words in the target text information to target real words; wherein the association relationship is obtained based on the phoneme features of words in the escaped text information and the real text information. The present disclosure can improve the recognition accuracy of escaped words.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically the field of natural language processing technology, and more particularly to a method, device, electronic device, and medium for processing text information, which can be applied to scenarios of identifying and modifying escaped words in text information. Background Art

[0002] A knowledge entity is the value of the intellectual labor embodied in intellectual assets. Its essence is intellectual conception, or creative thinking. For example, a knowledge entity could be a film or television work. In the current internet environment, there are cases where the names of knowledge entities are illegally disseminated by replacing them with pinyin or homophones without obtaining copyright.

[0003] Correcting the illegal dissemination behavior of knowledge entities is of great significance for protecting intellectual property rights, consolidating the foundation of innovation, and strengthening development momentum. Summary of the Invention

[0004] The present disclosure provides a method, device, electronic device, and medium for processing text information.

[0005] According to one aspect of the present disclosure, a method for processing text information is provided, the method comprising:

[0006] Get the target text information to be processed;

[0007] identifying a target escaped word in the target text information based on a predetermined association relationship between the candidate real word and the candidate escaped word, and modifying the target escaped word to restore the target escaped word in the target text information to a target real word;

[0008] The association relationship is obtained based on the phoneme features of words in the escaped text information and the real text information.

[0009] According to another aspect of the present disclosure, there is provided a device for processing text information, comprising:

[0010] A text information acquisition module is used to acquire target text information to be processed;

[0011] a text information processing module, configured to identify target escaped words in the target text information based on a predetermined association relationship between candidate real words and candidate escaped words, and modify the target escaped words to restore the target escaped words in the target text information to target real words;

[0012] The association relationship is obtained based on the phoneme features of words in the escaped text information and the real text information.

[0013] According to another aspect of the present disclosure, an electronic device is provided, the electronic device including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the text information processing method described in any embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the text information processing method described in any embodiment of the present disclosure.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the text information processing method described in any embodiment of the present disclosure when executed by a processor.

[0019] According to the technology of the present disclosure, the recognition accuracy of escape words can be improved.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flowchart of a method for processing text information provided according to an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of another method for processing text information provided according to an embodiment of the present disclosure;

[0024] Figure 3 is a flowchart of another method for processing text information provided according to an embodiment of the present disclosure;

[0025] Figure 4 is a flowchart of another method for processing text information provided according to an embodiment of the present disclosure;

[0026] Figure 5A is a flowchart of another method for processing text information provided according to an embodiment of the present disclosure;

[0027] Figure 5B is a flowchart of another method for processing text information provided according to an embodiment of the present disclosure;

[0028] Figure 6 is a structural diagram of a text information processing device provided according to an embodiment of the present disclosure;

[0029] Figure 7 It is a block diagram of an electronic device used to implement the text information processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] Figure 1 This is a flowchart of a text information processing method provided according to an embodiment of the present disclosure. The present disclosure embodiment is applicable to the situation where escaped words in text information are identified and modified in the scenario of correcting the behavior of the illegal dissemination of knowledge entities. The method can be executed by a text information processing device, which can be implemented in software and / or hardware and can be integrated into an electronic device that carries the text information processing function. Figure 1 As shown, the text information processing method of this embodiment may include:

[0032] S101, obtaining target text information to be processed;

[0033] S102, based on the predetermined association relationship between the candidate real words and the candidate escaped words, identifying the target escaped words in the target text information, and modifying the target escaped words to restore the target escaped words in the target text information to target real words; wherein the association relationship is obtained based on the phoneme features of the words in the escaped text information and the real text information.

[0034] The target text information to be processed refers to the text information that needs to be identified with escaped words. The target text information may or may not include escaped words. Whether the target text information includes escaped words depends on the actual situation and is not limited here.

[0035] Among them, escaped words are relative to real words, and escaped words are generally derived based on the pronunciation characteristics of real words.

[0036] Escape words are often used to circumvent compliance checks on the dissemination of target text information. It's well known that related technologies often rely on key words to check the compliance of target text information. Replacing actual words in the target text with escape words is a common tactic for circumventing these compliance checks.

[0037] Based on the association between candidate real words and candidate escaped words, it is possible to identify whether the target text information to be processed includes the target escaped word. The association is derived from the phonetic features of the words in the escaped text information and the real text information. The escaped text information describes the same object as the real text information, but the specific textual content in the escaped text information differs from that in the real text information. In other words, the real text information and the candidate text information express the same object differently.

[0038] Among them, the phoneme features are used to reflect the pronunciation characteristics of words in the real text information and the escaped text information. The real text information and the escaped text information are text corpora for building an association relationship. Optionally, the real text information is a conventional expression recognized by the industry. The escaped text information is mostly derived based on the pronunciation characteristics of the real text information, and the candidate real words and candidate escaped words are generated from the words in the real text information and the candidate text information. There is a correlation between the phoneme features of the candidate real words and the candidate escaped words. The three types of text information, the target text information, the real text information and the escaped text information, belong to the same field.

[0039] According to the predetermined association relationship between the candidate real words and the candidate escape words, the target escape words in the target text information are identified. Optionally, the words in the target text information are matched with the candidate escape words, and the candidate escape words appearing in the target text information are determined as the target escape words.

[0040] If the association relationship between the candidate real word and the candidate escape word is determined, then when the candidate escape word is known, the candidate real word associated with the candidate escape word can be determined based on the association relationship.

[0041] Optionally, a candidate real word associated with the target escape word is determined as the target real word.

[0042] When a target escape word is identified in the target text information, the target escape word is modified, and the target escape word in the target text information is restored using the target real word to restore the target escape word to the target real word.

[0043] This disclosed technical solution pre-establishes an association between candidate real words and candidate escaped words based on the phoneme features of the escaped text and the real text. Based on this association, the target escaped words in the target text are identified and then restored to the target real words. This improves the accuracy of escaped word recognition.

[0044] In an optional embodiment, the target text information to be processed is a knowledge entity name.

[0045] A knowledge entity is an entity that represents the value of intellectual labor within a knowledge asset. Its essence is intellectual conception, or creative thinking. For example, a knowledge entity might be a work of art or literature. The knowledge entity name is used to distinguish different knowledge entities. For example, if the knowledge entity is a film or television work, the knowledge entity name would be the title of the film or television work.

[0046] It is known that intellectual entities are protected by intellectual property rights, and the dissemination of intellectual entities requires the corresponding copyright. In the current internet environment, there are cases of illegal dissemination of intellectual entities by replacing the entity names with pinyin or homophones without obtaining copyright.

[0047] It is understandable that the names of knowledge entities are mostly creative words, and the words in the knowledge entity names lack contextual information. The disclosed technical solution predetermines the association between candidate real words and candidate escaped words based on the phonemic characteristics of the words, identifies the escaped words in the knowledge entity names based on this association, and restores the escaped words in the knowledge entity names to real words. This facilitates the correction of the illegal dissemination of knowledge entities and is conducive to improving the correction effect of the illegal dissemination of knowledge entities.

[0048] Figure 2 It is a flowchart of another method for processing text information provided according to an embodiment of the present disclosure; this embodiment is an optional scheme proposed on the basis of the above embodiment. Specifically, the embodiment of the present disclosure provides an association relationship construction scheme for pre-constructing the association relationship between the candidate real words and the candidate escape words.

[0049] See also Figure 2 The text information processing method provided in this embodiment includes:

[0050] S201 , performing word segmentation processing on the escaped text information and the real text information respectively to obtain the candidate real words and the candidate escaped words.

[0051] The escaped text information and the real text information are text corpora for establishing association relationships. The escaped text information and the real text information are used to generate candidate real words and candidate escaped words. Specifically, word segmentation processing is performed on the escaped text information and the real text information to obtain candidate real words and candidate escaped words.

[0052] S202: Determine the phoneme features of the candidate real words and the candidate escape words respectively.

[0053] Among them, phonemes are the smallest speech units divided according to the natural properties of speech. They are analyzed based on the pronunciation actions in syllables, and one action constitutes a phoneme.

[0054] The phoneme features are used to reflect the pronunciation characteristics of the candidate real words and the candidate escape words. Optionally, the mutually related candidate real words and the candidate escape words have the same pronunciation characteristics.

[0055] The association relationship between the candidate real words and the candidate escape words can be determined based on the phoneme features.

[0056] S203: Constructing an association relationship between the candidate real word and the candidate escape word based on the phoneme features of the candidate real word and the candidate escape word.

[0057] The phoneme features of the candidate real words are matched with the phoneme features of the candidate escape words, and based on the matching results between the two phoneme features, an association relationship between the candidate real words and the candidate escape words is constructed.

[0058] Optionally, based on the association between the candidate real words and the candidate escape words, mutually related candidate real words and candidate escape words are determined, and a confusion word set is constructed based on the mutually related candidate real words and candidate escape words. The escape words in the target text information are identified and modified based on the confusion word set.

[0059] The technical solution disclosed in the present invention obtains candidate real words and candidate escaped words by performing word segmentation processing on escaped text information and real text information, constructs an association relationship between the candidate real words and the candidate escaped words based on the phonemic features of the candidate real words and the candidate escaped words, identifies and modifies the escaped words in the target text information based on the association relationship, and improves the recognition accuracy of the escaped words.

[0060] Figure 3 This is a flowchart of another method for processing text information according to an embodiment of the present disclosure. This embodiment is an optional solution proposed based on the above embodiment. Specifically, this embodiment of the present disclosure refines the operation of "respectively determining the phonemic features of the candidate real words and the candidate escape words."

[0061] See also Figure 3 The text information processing method provided in this embodiment includes:

[0062] S301 , performing word segmentation processing on the escaped text information and the real text information respectively to obtain the candidate real words and the candidate escaped words.

[0063] S302, determining the pinyin syllables and pinyin abbreviations of the candidate real words and the candidate escape words.

[0064] A syllable is a language unit consisting of one or more phonemes. Pinyin syllables are used to determine the pronunciation of words, and Pinyin refers to the process of spelling syllables.

[0065] Pinyin abbreviations are determined based on Pinyin syllables. Specifically, they are obtained by discarding some phonemes within a Pinyin syllable. Alternatively, the first phoneme in each Pinyin syllable is retained, the remaining phonemes are discarded, and the remaining phonemes are arranged in a pre-determined order to obtain the Pinyin abbreviation. Compared to Pinyin syllables, Pinyin abbreviations are somewhat more concealed.

[0066] S303: Determine the phoneme features of the candidate real words and the candidate escape words according to the pinyin syllables and the pinyin abbreviations.

[0067] Both pinyin syllables and pinyin abbreviations can reflect the pronunciation characteristics of a word. Optionally, the pinyin syllables and pinyin abbreviations of the candidate real word are concatenated, and the resulting concatenated string is determined as the phonemic feature of the candidate real word; the pinyin syllables and pinyin abbreviations of the candidate escape word are concatenated, and the resulting concatenated string is determined as the phonemic feature of the candidate escape word.

[0068] S304: Constructing an association relationship between the candidate real word and the candidate escape word based on the phoneme features of the candidate real word and the candidate escape word.

[0069] Optionally, the phoneme features of the candidate real words are matched with the phoneme features of the candidate escape words, and an association relationship between the candidate real words and the candidate escape words is constructed based on the matching results of the phoneme features.

[0070] The disclosed embodiment determines the pinyin syllables and pinyin abbreviations of candidate real words and candidate escaped words, determines phoneme features based on the pinyin syllables and pinyin abbreviations, and uses the phoneme features to determine the association relationship between the candidate real words and the candidate escaped words. This can cover the situation where homophonic words, pinyin syllables or pinyin abbreviations are used as candidate escaped words, which is beneficial to improving the recognition accuracy of escaped words.

[0071] In an optional embodiment, based on the phoneme features of the candidate real words and the candidate escape words, an association relationship between the candidate real words and the candidate escape words is constructed, including: determining the edit distance between the phoneme features of any candidate real word and the phoneme features of any candidate escape word; and constructing the association relationship between the candidate real words and the candidate escape words based on the edit distance.

[0072] Optionally, the data type of the phoneme feature is a string. An edit distance between the phoneme features is determined, and an association relationship between the candidate real word and the candidate escape word is determined based on the edit distance.

[0073] The edit distance measures the degree of match between the phoneme features of a candidate real word and the phoneme features of a candidate escape word. The edit distance is a quantitative measure of the difference between two phoneme features. It is measured by determining the minimum number of operations required to transform one string into another.

[0074] Optionally, an edit distance is determined between the phoneme features of any candidate real word and the phoneme features of any candidate escape word. Based on the relative magnitude of the edit distance and a preset distance threshold, mutually associated candidate real words and candidate escape words are determined, thereby obtaining an association relationship between the candidate real words and candidate escape words. The preset distance threshold is determined based on actual circumstances and is not limited here.

[0075] The above technical solution determines the edit distance between the phoneme features of the candidate real words and the phoneme features of the candidate escaped words, uses the edit distance to quantify the similarity between the candidate real words and the candidate escaped words, and constructs the association relationship between the candidate real words and the candidate escaped words based on the edit distance between different phoneme features, providing a practical and feasible association relationship construction scheme, which is conducive to improving the recognition accuracy of escaped words.

[0076] Figure 4 This is a flowchart of another method for processing text information according to an embodiment of the present disclosure; this embodiment is an optional solution based on the above embodiment. Specifically, this embodiment of the present disclosure refines the operation of "separately segmenting the escaped text information and the real text information to obtain the candidate real words and the candidate escaped words."

[0077] See also Figure 4 The text information processing method provided in this embodiment includes:

[0078] S401 , performing word segmentation processing on the escape text information and the real text information according to the word types of the words in the escape text information and the real text information to obtain the candidate real words and candidate escape words.

[0079] The escaped text information and the real text information may include multiple different types of words at the same time. Optionally, the word type of the words in the escaped text information and the real text information is determined based on whether the words in the escaped text information and the real text information are Chinese words. The words in the escaped text information and the real text information are divided into Chinese words and other words other than Chinese words. Exemplarily, the other words may be English words.

[0080] S402: Determine the phoneme features of the candidate real words and the candidate escape words respectively.

[0081] S403: Constructing an association relationship between the candidate real word and the candidate escape word based on the phoneme features of the candidate real word and the candidate escape word.

[0082] In this disclosed embodiment, escaped text information and real text information are segmented based on word type to obtain candidate real words and candidate escaped words. This provides technical support for subsequently determining the association between the candidate real words and candidate escaped words, and using the association to identify escaped words in the target text information.

[0083] In an optional embodiment, the escape text information and the real text information are respectively segmented according to the word types of the words in the escape text information and the real text information, including: segmenting the Chinese words in the escape text information and the real text information at a single-word granularity; performing a grammar check on the words other than Chinese words in the escape text information and the real text information, and segmenting the words other than Chinese words based on the grammar check results.

[0084] Determine the types of words in the escaped text information and the real text information. Specifically, determine whether the words in the escaped text information and the real text information are Chinese words or words other than Chinese words.

[0085] The Chinese words in the escaped text information and the real text information are segmented based on the single-word dimension. A grammar check is performed on the words other than the Chinese words in the escaped text information and the real text information. Optionally, the grammar check includes an English grammar check and a pinyin grammar check. Exemplarily, an English grammar check can be performed on the escaped text information and the real text information first. If the English grammar check is passed, the other words in the escaped text information and the real text information are segmented based on the English grammar. If the English grammar check is not passed, a pinyin grammar check can be performed on the other words in the escaped text information and the real text information. If the pinyin grammar check is passed, the other words are segmented based on the pinyin grammar.

[0086] Optionally, other words in the escaped text information and the real text information that fail the English grammar check and the pinyin grammar check are segmented at a single-word granularity.

[0087] The above technical solution segments the Chinese words in the real text information and the escaped text information at the single-word granularity, performs grammar check on the words other than Chinese words in the escaped text information and the real text information, and segments the words other than Chinese words based on the grammar check results to obtain candidate real words and candidate escaped words, thereby ensuring the integrity of the candidate real words and candidate escaped words, providing data support for the subsequent determination of the association relationship between the candidate real words and the candidate escaped words, and facilitating the accuracy of the association relationship.

[0088] Figure 5A This is a flowchart of another method for processing text information provided in accordance with an embodiment of the present disclosure; this embodiment is an optional solution proposed based on the above embodiment. Specifically, this embodiment of the present disclosure refines the operation of "identifying target escaped words in the target text information based on the predetermined association relationship between candidate real words and candidate escaped words, modifying the target escaped words, and restoring the target escaped words in the target text information to target real words."

[0089] See also Figure 5A The text information processing method provided in this embodiment includes:

[0090] S501: Obtain target text information to be processed.

[0091] S502: Perform word segmentation processing on the target text information according to the word types of the words in the target text information to obtain target questionable words to be processed.

[0092] The target text information may include multiple different types of words at the same time. Optionally, the word type of the words in the target text information is determined based on whether the words in the target text information are Chinese words, and the words in the target text information are divided into Chinese words and other words other than Chinese words. Exemplarily, the other words may be English words.

[0093] According to the word types of the words in the target text information, the target text information is segmented using the corresponding word segmentation method to obtain the target questionable words to be processed.

[0094] The target questionable words refer to words in the target text information. The target questionable words can be obtained by performing word segmentation on the target text information. Whether the target questionable words are escape words has not yet been determined.

[0095] S503: Determine the target questionable word that matches the candidate escape word as the target escape word.

[0096] Optionally, the candidate real words and the candidate escaped words are obtained by performing word segmentation processing on the real text information and the escaped text information according to word types.

[0097] The target questionable word is matched with the candidate escape word, and the target questionable word that matches the candidate escape word is determined as the target escape word.

[0098] S504: Based on the predetermined association relationship between the candidate real words and the candidate escape words, determine the candidate real word associated with the target escape word as a target real word.

[0099] The association relationship is obtained based on the phoneme features of words in the escaped text information and the real text information.

[0100] If the association relationship between the candidate real word and the candidate escape word is determined, then when the candidate escape word is known, the candidate real word associated with the candidate escape word can be determined based on the association relationship.

[0101] The candidate real words associated with the target escape word are determined as the target real words.

[0102] S505: Modify the target questionable word based on the target real word, and restore the target escaped word in the target text information to the target real word.

[0103] The target real words associated with the target escape words are used to modify the target questionable words, and the target escape words in the target text information are restored to the target real words.

[0104] In an embodiment of the present disclosure, by performing word segmentation on the target text information to be processed, the target suspicious words to be processed are obtained. Based on the pre-determined association relationship between the candidate real words and the candidate escape words, it is determined whether the target suspicious words are escape words; in the case where the target suspicious words are target escape words, the target suspicious words are modified using the target real words associated with the target escape words, which improves the recognition accuracy of the escape words.

[0105] In a specific embodiment, the target text information is the name of a knowledge entity. For the sake of easy understanding, taking the name of a knowledge entity as the name of a film or television work as an example, the processing method of the text information provided by the embodiment of the present disclosure is introduced.

[0106] First, the construction process of the association relationship between the candidate real words and the candidate escape words is introduced. Figure 5B It is a flowchart of another processing method of text information provided by an embodiment of the present disclosure. Refer to Figure 5B , Figure 5B A specific method for constructing the association relationship is provided.

[0107] The target text information, the real text information, and the escape text information belong to the same field. In the case where the target text information is the name of a film or television work, the real text information and the escape text information are also the names of film or television works. The real text information includes the real name of the work, and the real work name corresponds to the candidate real words. The escape text information includes the escape name of the work, and the work escape name corresponds to the candidate escape words. Optionally, the work escape name is derived based on the pronunciation characteristics of the real name of the work. The work escape name can be obtained based on the homophonic words, pinyin syllables, or pinyin abbreviations of the real name of the work. Exemplarily, the real text information is "Warriors of Future Past", and the corresponding escape text information is "Warriors of Future Past yn", "jzh yunie", or "jinzhi Future Past", etc.

[0108] The real text information and the escape text information are the text corpora for constructing the association relationship. Optionally, data cleaning is performed on the real text information and the escape text information, and word segmentation is performed on the cleaned real text information and escape text information according to the word type. It is not known in advance which text information in the text corpus is the real text information including the candidate real words and which is the escape text information including the candidate escape words. Optionally, any one of the text corpora for constructing the association relationship is selected as the seed text information, and word segmentation is performed on the seed text information and the other text information in the text corpus except the seed text information according to the word type. That is, it is not certain whether the seed text information is the real text information.

[0109] Optionally, the seed text information and the other text information in the text corpus except the seed text information are segmented according to the word type.

[0110] Specifically, first, the Chinese text information of the seed text and other text information is checked. The Chinese words in the seed text information and the Chinese words in the other text information in the text corpus except the seed text information are segmented at the single-character level. The grammar of other words in the seed text information and other text information is checked. Among them, the grammar check includes English grammar check and Chinese grammar check. Optionally, first, the English grammar of other words in the seed text information and other text information is checked. If the English grammar check passes, the word is segmented according to English grammar; if the English grammar check fails, the phonetic grammar of the word is checked. If the phonetic grammar check passes, the word is segmented according to phonetic grammar. If the word fails both the English grammar check and the phonetic grammar check, the word is segmented at the single-character level.

[0111] When the seed text information is "Golden Branch yn" and one of the other text information is "Warriors of Future Past", the Chinese words in the seed text information and this other text information are segmented at the single-character level. "Golden Branch" in the seed text information is segmented into "Gold" and "Branch". "Warriors of Future Past" in the other text information is segmented into "Desire" and "Evil". Then, the grammar of other words in the seed text information and other text information is checked. For example, the English grammar of "yn" and "jinzhi" in the seed text information is checked first. Both "yn" and "jinzhi" fail the English grammar check. Then, the phonetic grammar of "yn" and "jinzhi" is checked. "jinzhi" passes the phonetic grammar check and is segmented into "jin" and "zhi". "yn" fails both the phonetic grammar check and the English grammar check and is segmented at the single-character level into "y" and "n".

[0112] Optionally, before checking the grammar of other words in the seed text information and other text information except Chinese words, the character length of this other word is determined. The character length is compared with a preset character length. When the character length is greater than the preset character length, the grammar of this other word is checked. The preset character length is determined according to the actual situation and is not limited here. Exemplarily, the preset character length is 2. This can reduce the occupation of computing resources and improve the efficiency of text information processing.

[0113] By performing word segmentation on the seed text information and all other text information, a word segmentation result can be obtained, and the word segmentation result includes candidate real words and candidate escape words.

[0114] Optionally, before splitting the seed text information and other text information in the text corpus except the seed text information according to the word type, split the seed text information and other text information except the seed text information at the string dimension to obtain the to-be-processed word chunks with complete semantics. Exemplarily, splitting the text information "I love super meiweiDonut" at the string dimension can obtain the to-be-processed word chunks including: "I", "love", "super", and "meiweiDonut". The Chinese words "I love super" in the text information have clear semantics and can be split into 3 to-be-processed word chunks "I", "love", and "super". The other words "meiweiDonut" in the text information except the Chinese words have unclear semantics and are taken as 1 to-be-processed word chunk as a whole. Then, perform fine-grained word segmentation processing on the to-be-processed word chunks according to the word type based on the above method. This can ensure the semantic integrity of the to-be-processed word chunks and is beneficial to improving the recognition accuracy of escaped words.

[0115] After obtaining the word segmentation results of the seed text information and other text information, process the word segmentation results obtained by using the n-gram algorithm, and extract words of the same length from the word segmentation results belonging to the seed text information and other text information. Among them, n-gram is an algorithm based on a statistical language model. Its basic idea is to perform a sliding window operation of size n on the content in the text according to bytes, forming a sequence of byte segments of length n. Among them, n is a positive integer. In the n-gram algorithm of the embodiments of the present disclosure, it is used to perform word extraction processing from the word segmentation results.

[0116] The words extracted from the word segmentation results of the seed text information are represented by seed_n_gram, and the words extracted from the word segmentation results of other text information are represented by S2_n_gram. When the n-gram algorithm is a 2-gram algorithm, the word lengths of seed_n_gram and S2_n_gram are 2. Continuing with the above example, seed_n_gram can be [gold, branch], [branch, y], and [y, n], etc.; S2_n_gram can be [jin, zhi], [zhi, desire], [desire, evil].

[0117] Optionally, the word segmentation results processed by using the n-gram algorithm include candidate real words and candidate escaped words.

[0118] Determine the phoneme features of seed_n_gram and S2_n_gram. Specifically, convert seed_n_gram and S2_n_gram into pinyin syllables respectively, and determine the pinyin abbreviations of seed_n_gram and S2_n_gram. Specifically, convert the Chinese words in seed_n_gram and S2_n_gram into pinyin syllables, and determine the pinyin abbreviations of these Chinese words. For other words in seed_n_gram and S2_n_gram that pass the pinyin grammar check, directly retain them and determine their pinyin abbreviations. For other words in seed_n_gram and S2_n_gram that pass the English grammar check or fail both grammar checks, directly retain them. Continuing with the above example, when seed_n_gram includes [金, 枝] and [枝, y], the phoneme features of [金, 枝] can be determined as [金, jin, j, 枝, zhi, zh]; the phoneme features of [枝, y] are [枝, zhi, zh, y]. When S2_n_gram includes [jin, zhi] and [zhi, 欲], the phoneme features of [jin, zhi] are [jin, j, zhi, zh]; the phoneme features of [zhi, 欲] are [zhi, zh, 欲, yu, y].

[0119] Then, calculate the edit distance between the phoneme features of seed_n_gram and the phoneme features of S2_n_gram respectively, compare the edit distance with the preset edit distance, and determine the associated candidate real words and candidate escape words from seed_n_gram and S2_n_gram according to the comparison result to form a pair of associated words. Optionally, select pairs of associated words including at least one pair of all Chinese words to construct a set of confusing words, and discard the associated words including only other words. Use the word association relationship recorded in the pairs of associated words included in the set of confusing words as the association relationship between the candidate real words and the candidate escape words. Continuing with the above example, the pairs of associated words in the set of confusing words can include: [金枝] and [jin, zhi], and [yn] and [欲孽], and [玉聂] and [欲孽], etc. Among them, the set of confusing words is used to identify the escape words in the target text information.

[0120] Then, introduce the method of identifying the escape words in the target text information based on the association relationship between the candidate real words and the candidate escape words.

[0121] Specifically, according to the word types of the words in the target text information, the words in the target text information are segmented to obtain a word segmentation result. According to the lengths of the words that form the error correction word pairs in the confusion word set, words are taken from the word segmentation result of the target text information to obtain the target suspicious words to be processed. Then, the target suspicious words are matched with the error correction word pairs in the confusion word set to identify the target escape words in the target text information, where the target escape words belong to the confusion word set. The candidate real words associated with the target escape words in the confusion word set are determined as the target real words, and the target escape words are replaced with the target real words. Exemplarily, the error correction word pairs included in the confusion word set include: [Water Margin] and [shuihu], and [Water Margin] and [s Lake], etc. When the target text information is "s Lake Biography", based on the confusion word set, the escape word "s Lake" in the target text information can be identified, and the target real word "Water Margin" is used to restore "s Lake", so that the restored target text information is "Water Margin". Optionally, the compliance of the restored target text information dissemination is checked.

[0122] Figure 6 It is a structural schematic diagram of a text information processing device provided by an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to the situation of identifying and modifying escape words in text information in the scenario of correcting the behavior of illegally spreading knowledge entities. This device can be implemented by software and / or hardware, and this device can implement the text information processing method described in any embodiment of the present disclosure. As Figure 6 shown, the text information processing device 600 includes:

[0123] A text information acquisition module 601, configured to acquire target text information to be processed;

[0124] A text information processing module 602, configured to identify target escape words in the target text information according to the association relationship between candidate real words and candidate escape words determined in advance, and modify the target escape words, and restore the target escape words in the target text information to target real words;

[0125] Wherein, the association relationship is obtained according to the phoneme characteristics of the words in the escape text information and the real text information.

[0126] The technical solution of the present disclosure pre-constructs the association relationship between candidate real words and candidate escape words according to the phoneme characteristics of the escape text information and the real text information, and based on this association relationship, identifies the target escape words in the target text information, and restores the target escape words in the target text information to target real words. The recognition accuracy of escape words is improved.

[0127] Optionally, the device further includes: an association relationship building module for pre-building an association relationship between the candidate real words and the candidate escaped words; the association relationship building module includes: a word segmentation processing submodule for performing word segmentation processing on the escaped text information and the real text information respectively to obtain the candidate real words and the candidate escaped words; a phoneme feature determination submodule for determining the phoneme features of the candidate real words and the candidate escaped words respectively;

[0128] The association relationship construction submodule is used to construct an association relationship between the candidate real words and the candidate escape words based on the phoneme features of the candidate real words and the candidate escape words.

[0129] Optionally, the association relationship construction submodule includes: an edit distance determination unit, used to determine the edit distance between the phoneme features of any candidate real word and the phoneme features of any candidate escape word; and an association relationship construction unit, used to construct the association relationship between the candidate real word and the candidate escape word based on the edit distance.

[0130] Optionally, the phoneme feature determination submodule includes: a pinyin data determination unit, configured to determine the pinyin syllables and pinyin abbreviations of the candidate real words and the candidate escape words;

[0131] A phoneme feature determination unit is used to determine the phoneme features of the candidate real words and the candidate escape words based on the pinyin syllables and the pinyin abbreviations.

[0132] Optionally, a word segmentation processing submodule is specifically used to perform word segmentation processing on the escape text information and the real text information according to the word types of the words in the escape text information and the real text information to obtain the candidate real words and candidate escape words.

[0133] Optionally, the word segmentation processing submodule includes: a first word segmentation processing unit, used to segment the Chinese words in the escape text information and the real text information at a single-word granularity; a second word segmentation processing unit, used to perform a grammar check on the words other than Chinese words in the escape text information and the real text information, and segment the words other than Chinese words based on the grammar check results.

[0134] Optionally, the text information processing module 602 includes: a questionable word determination submodule, which is used to perform word segmentation processing on the target text information according to the word type of the words in the target text information to obtain the target questionable words to be processed; an escaped word identification submodule, which is used to determine the target questionable words that match the candidate escaped words as the target escaped words; a real word determination submodule, which is used to determine the candidate real words associated with the target escaped words as the target real words based on the association relationship between the predetermined candidate real words and the candidate escaped words; and a questionable word modification submodule, which is used to modify the target questionable words based on the target real words, and restore the target escaped words in the target text information to the target real words.

[0135] Optionally, the target text information to be processed is a knowledge entity name.

[0136] The text information processing device provided by the embodiments of the present disclosure can execute the text information processing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of executing the text information processing method.

[0137] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0138] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0139] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0141] Multiple components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0142] The computing unit 701 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the text information processing method. For example, in some embodiments, the text information processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the text information processing method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the text information processing method by any other appropriate means (e.g., by means of firmware).

[0143] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable text information processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0147] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0148] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0149] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0150] Cloud computing refers to a technology system that provides network access to elastically scalable shared pools of physical or virtual resources. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on-demand in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as for model training.

[0151] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0152] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for processing text information, the method comprising: Get the target text information to be processed; identifying a target escaped word in the target text information based on a predetermined association relationship between the candidate real word and the candidate escaped word, and modifying the target escaped word to restore the target escaped word in the target text information to a target real word; Among them, the association relationship is obtained based on the phonemic features of the words in the escape text information and the real text information; the escape text information and the real text information are used to generate the candidate real words and the candidate escape words; the association relationship is constructed based on the phonemic features of the candidate real words and the candidate escape words; the phonemic features of the candidate real words and the candidate escape words are determined based on pinyin syllables and pinyin abbreviations.

2. The method according to claim 1, wherein The association relationship between the candidate real words and the candidate escape words is pre-constructed based on the following method: Performing word segmentation processing on the escaped text information and the real text information respectively to obtain the candidate real words and the candidate escaped words; determining the phoneme features of the candidate real words and the candidate escape words respectively; Based on the phoneme features of the candidate real words and the candidate escape words, an association relationship between the candidate real words and the candidate escape words is constructed.

3. The method according to claim 2, wherein: Constructing an association relationship between the candidate real word and the candidate escape word based on the phoneme features of the candidate real word and the candidate escape word, including: Determining the edit distance between the phoneme features of any candidate real word and the phoneme features of any candidate escape word; According to the edit distance, an association relationship between the candidate real word and the candidate escape word is constructed.

4. The method according to claim 2, wherein: The determining of the phoneme features of the candidate real words and the candidate escape words respectively includes: Determining the pinyin syllables and pinyin abbreviations of the candidate real words and the candidate escape words; The phoneme features of the candidate real words and the candidate escape words are determined according to the pinyin syllables and the pinyin abbreviations.

5. The method according to claim 2, wherein: Performing word segmentation processing on the escaped text information and the real text information respectively to obtain the candidate real words and the candidate escaped words, including: According to the word types of the words in the escape text information and the real text information, word segmentation processing is performed on the escape text information and the real text information respectively to obtain the candidate real words and candidate escape words.

6. The method according to claim 5, wherein: Performing word segmentation processing on the escaped text information and the real text information respectively according to word types of words in the escaped text information and the real text information, including: Segmenting the Chinese words in the escaped text information and the real text information at a single-word granularity; Grammar checking is performed on the words other than Chinese words in the escaped text information and the real text information, and the words other than Chinese words are segmented based on the grammar checking result.

7. The method according to claim 1, wherein According to a predetermined association relationship between candidate real words and candidate escaped words, a target escaped word in the target text information is identified, and the target escaped word is modified to restore the target escaped word in the target text information to a target real word, including: Performing word segmentation processing on the target text information according to the word type of the words in the target text information to obtain target questionable words to be processed; Determining the target questionable word that matches the candidate escape word as the target escape word; Based on the predetermined association relationship between the candidate real words and the candidate escape words, determining the candidate real word associated with the target escape word as a target real word; The target questionable word is modified based on the target real word, and the target escaped word in the target text information is restored to the target real word.

8. The method according to any one of claims 1 to 7, wherein: The target text information to be processed is the name of a knowledge entity.

9. A text information processing device, comprising: A text information acquisition module is used to acquire target text information to be processed; a text information processing module, configured to identify target escaped words in the target text information based on a predetermined association relationship between candidate real words and candidate escaped words, and modify the target escaped words to restore the target escaped words in the target text information to target real words; Among them, the association relationship is obtained based on the phonemic features of the words in the escape text information and the real text information; the escape text information and the real text information are used to generate the candidate real words and the candidate escape words; the association relationship is constructed based on the phonemic features of the candidate real words and the candidate escape words; the phonemic features of the candidate real words and the candidate escape words are determined based on pinyin syllables and pinyin abbreviations.

10. The device according to claim 9, wherein The apparatus further comprises: an association relationship building module for pre-building an association relationship between the candidate real words and the candidate escape words; The association relationship building module includes: a word segmentation processing submodule, which is used to perform word segmentation processing on the escaped text information and the real text information respectively to obtain the candidate real words and the candidate escaped words; a phoneme feature determination submodule, configured to determine the phoneme features of the candidate real words and the candidate escape words respectively; The association relationship construction submodule is used to construct an association relationship between the candidate real words and the candidate escape words based on the phoneme features of the candidate real words and the candidate escape words.

11. The device according to claim 10, wherein The association relationship building sub-module includes: an edit distance determining unit, configured to determine an edit distance between a phoneme feature of any candidate real word and a phoneme feature of any candidate escape word; An association relationship building unit is used to build an association relationship between the candidate real word and the candidate escape word according to the edit distance.

12. The device according to claim 10, wherein The phoneme feature determination submodule includes: a pinyin data determining unit, configured to determine the pinyin syllables and pinyin abbreviations of the candidate real words and the candidate escaped words; A phoneme feature determination unit is used to determine the phoneme features of the candidate real words and the candidate escape words based on the pinyin syllables and the pinyin abbreviations.

13. The device according to claim 10, wherein The word segmentation processing submodule is specifically used to perform word segmentation processing on the escape text information and the real text information according to the word types of the words in the escape text information and the real text information to obtain the candidate real words and candidate escape words.

14. The device according to claim 13, wherein The word segmentation processing submodule includes: a first word segmentation processing unit, configured to segment the Chinese words in the escaped text information and the real text information at a single-word granularity; The second word segmentation processing unit is configured to perform a grammar check on the words other than Chinese words in the escaped text information and the real text information, and segment the words other than Chinese words based on the grammar check result.

15. The device according to claim 9, wherein Text information processing module, including: a doubtful word determination submodule, configured to perform word segmentation processing on the target text information according to the word type of the words in the target text information to obtain target doubtful words to be processed; an escape word identification submodule, configured to determine the target questionable word that matches the candidate escape word as the target escape word; a real word determination submodule, configured to determine the candidate real word associated with the target escaped word as a target real word based on the association relationship between the predetermined candidate real word and the candidate escaped word; The doubtful word modification submodule is used to modify the target doubtful word based on the target real word, and restore the target escaped word in the target text information to the target real word.

16. The device according to any one of claims 9 to 15, wherein: The target text information to be processed is the name of a knowledge entity.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to execute the text information processing method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the text information processing method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the text information processing method according to any one of claims 1 to 8.

Images