Text correction method and device, equipment and computer readable medium

Abstract

The application relates to a text error correction method, device and equipment and a computer readable medium. The method comprises the following steps: obtaining a text to be corrected; identifying the text to be corrected based on a first knowledge graph to obtain a target wrong word in the text to be corrected, the target wrong word being a word irrelevant to the content of the text to be corrected, and the first knowledge graph being used for recording domain knowledge related to the content of the text to be corrected; determining a candidate replacement word of the target wrong word based on a second knowledge graph, the second knowledge graph being used for recording domain knowledge of easily confused texts; performing feature sorting on a candidate text obtained by replacing the target wrong word with the candidate replacement word; and determining the optimal text of the feature sorting as a corrected text. The application solves the technical problem of low text error correction accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field, and in particular to a text error correction method, device and equipment and computer readable medium. BACKGROUND

[0002] The Chinese error correction technology is an important technology for realizing automatic checking and automatic correction of Chinese sentences, and aims to improve language correctness while reducing manual checking cost. The error correction module is the most basic module of natural language processing, and its importance is self-evident.

[0003] In daily life, we often find many wrong words in social tools or public number articles. According to research, the text error probability in new media fields such as microblog is about 2%, and the error rate in the speech recognition field can be as high as 8-10%.

[0004] At present, in the related technology, the text error correction is mainly based on pinyin. First, the pinyin of all local dictionaries is placed in a data structure, and then the pinyin of the text to be corrected is obtained by annotating the text to be corrected. The edit distance of the pinyin of the text to be corrected and the pinyin of all dictionaries is calculated, a certain threshold is set, the text with the closest edit distance of the pinyin of the text to be corrected is obtained as the candidate text, and the text with the highest score is selected by a certain rule. For a long text, the edit distance based on the word is used to calculate the edit distance based on the word between the input text to be corrected and the existing dictionary text, and then the text with the highest score is calculated by a certain sorting rule. Whether the text is corrected based on the pinyin or the edit distance based on the word, it is very dependent on the dictionary. Once the user inputs the text that does not exist in the dictionary, the error correction accuracy is very low, which causes a very poor user experience for the user.

[0005] At present, there is no effective solution to the technical problem of low text error correction accuracy. SUMMARY

[0006] The present application provides a text error correction method, device, equipment and computer readable medium to solve the technical problem of low text error correction accuracy.

[0007] According to one aspect of an embodiment of the present application, the present application provides a text error correction method, comprising:

[0008] obtaining a text to be corrected;

[0009] identifying the text to be corrected based on a first knowledge graph to obtain a target wrong word in the text to be corrected, the target wrong word being a word in the text to be corrected that is not related to the content of the text to be corrected, and the first knowledge graph being used to record domain knowledge related to the content of the text to be corrected;

[0010] determine a candidate replacement word for the target wrong word based on a second knowledge graph, the second knowledge graph being used to record domain knowledge of the confusing text;

[0011] perform feature sorting on the candidate text obtained by replacing the target wrong word with the candidate replacement word;

[0012] determine the optimal text of the feature sorting as the corrected text.

[0013] Optionally, the target wrong word in the text to be corrected is determined based on the first knowledge graph, comprising:

[0014] extract a target dictionary from the first knowledge graph, and pre-process the text to be corrected, the content of the target dictionary comprising target attribute values of corresponding entities in the first knowledge graph, the pre-processing comprising at least one of format conversion, format standardization and character filtering;

[0015] split the pre-processed text to be corrected into a plurality of character combinations using the target dictionary, and match each character combination with data in the target dictionary, wherein each character is assigned to at least one character combination, and each character combination comprises at least one character;

[0016] determine the character combination that does not match the data in the target dictionary as the target wrong word.

[0017] Optionally, the target wrong word in the text to be corrected is determined based on the first knowledge graph, further comprising:

[0018] create a dictionary tree using the first knowledge graph, the dictionary tree being used to store a mapping relationship from pinyin to entity of domain knowledge related to the content of the text to be corrected, each node in the dictionary tree corresponding to storage of a pinyin character in the pinyin;

[0019] convert the character combination into a pinyin combination, and determine a target pinyin chain with the smallest edit distance to the pinyin combination in the dictionary tree;

[0020] determine a preset character combination of the entity corresponding to the target pinyin chain according to the mapping relationship from pinyin to entity;

[0021] in the case where the character combination is inconsistent with the preset character combination, determine the character combination as the target wrong word.

[0022] Optionally, the target wrong word in the text to be corrected further comprises:

[0023] cover each character combination in the text to be corrected in sequence according to the order of the text in the text to be corrected, to obtain a plurality of covered text sequences;

[0024] input the covered text sequence into a mask language model, and obtain predicted characters and corresponding prediction probabilities output by the mask language model, the predicted characters being characters filling the covered regions identified by the mask language model according to context semantics of the covered characters, and the prediction probabilities being probabilities of the predicted characters filling the covered regions;

[0025] In a case where the combination of the covered characters is inconsistent with the corresponding predicted characters, the covered characters are determined as the target wrong word.

[0026] Optionally, determining the candidate replacement word of the target wrong word based on the second knowledge graph comprises:

[0027] extracting a confusion word data set from the second knowledge graph;

[0028] determining, as the candidate replacement word, a word in the confusion word data set that has a similarity greater than or equal to a similarity threshold with the target wrong word.

[0029] Optionally, determining the candidate replacement word of the target wrong word further comprises at least one of the following manners:

[0030] determining a preset character combination as the candidate replacement word;

[0031] In a case where the prediction probability of the predicted character is greater than or equal to a probability threshold, determining the predicted character as the candidate replacement word.

[0032] Optionally, performing feature sorting on the candidate text obtained by replacing the target wrong word with the candidate replacement word comprises:

[0033] inputting the candidate text into a logistic regression model, and obtaining a text sorting result output by the logistic regression model, the logistic regression model being used to extract text features and perform feature sorting, and the text sorting result being a similarity sorting result of the text features of the candidate text and content features of the text to be corrected;

[0034] The text features comprise at least one of the following:

[0035] a selection frequency of the candidate text;

[0036] an edit distance between the candidate text and the text to be corrected;

[0037] a Jaccard distance between pinyin of the candidate text and pinyin of the text to be corrected;

[0038] a semantic accuracy degree of the candidate text determined by a multilingual model.

[0039] According to another aspect of the embodiments of the present application, the present application provides a text correction device, comprising:

[0040] a text acquisition module configured to acquire a text to be corrected;

[0041] an error detection module configured to identify the to-be-corrected text based on the first knowledge graph to obtain a target error word in the to-be-corrected text, the target error word being a word in the to-be-corrected text that is irrelevant to the content of the to-be-corrected text, and the first knowledge graph being configured to record domain knowledge related to the content of the to-be-corrected text;

[0042] a candidate recall module configured to determine a candidate replacement word of the target error word based on a second knowledge graph, the second knowledge graph being configured to record domain knowledge of confusing texts;

[0043] a candidate sorting module configured to sort features of a candidate text obtained by replacing the target error word with the candidate replacement word;

[0044] a correction module configured to determine an optimal text of the feature sorting as a corrected text.

[0045] According to another aspect of the embodiments of the present application, the present application provides an electronic device including a memory, a processor, a communication interface and a communication bus, the memory storing a computer program executable on the processor, the memory, the processor and the communication interface communicating through the communication bus, and the processor executing the computer program to implement the steps of the above method.

[0046] According to another aspect of the embodiments of the present application, the present application further provides a computer readable medium having non-volatile program codes executable by a processor, the program codes causing the processor to execute the above method.

[0047] The present solution can be applied to the field of deep learning for natural language processing, and the above technical solution provided by the embodiments of the present application has the following advantages compared with related technologies.

[0048] The technical solution of the present application is to obtain a to-be-corrected text; identify the to-be-corrected text based on a first knowledge graph to obtain a target error word in the to-be-corrected text, the target error word being a word in the to-be-corrected text that is irrelevant to the content of the to-be-corrected text, and the first knowledge graph being configured to record domain knowledge related to the content of the to-be-corrected text; determine a candidate replacement word of the target error word based on a second knowledge graph, the second knowledge graph being configured to record domain knowledge of confusing texts; sort features of a candidate text obtained by replacing the target error word with the candidate replacement word; and determine an optimal text of the feature sorting as a corrected text. The present application combines knowledge graphs to perform text correction in the manner of error detection, candidate recall and candidate sorting, obtains a dictionary of domain knowledge and a confusing data set by mining the knowledge graph, and increases the recall of candidate words. The result of text correction is more accurate, and the technical problem of low accuracy of text correction is solved. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings, which are incorporated herein and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the accompanying drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0051] Figure 1 A hardware environment schematic diagram of an optional text correction method according to the embodiments of the present application is provided.

[0052] Figure 2 A flowchart of an optional text correction method according to the embodiments of the present application is provided.

[0053] Figure 3 An optional error detection schematic diagram according to the embodiments of the present application is provided.

[0054] Figure 4 An optional candidate recall schematic diagram according to the embodiments of the present application is provided.

[0055] Figure 5 An optional candidate sorting schematic diagram according to the embodiments of the present application is provided.

[0056] Figure 6 A block diagram of an optional text correction device according to the embodiments of the present application is provided.

[0057] Figure 7 An optional electronic device structure schematic diagram according to the embodiments of the present application is provided. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the scope of protection of the present application.

[0059] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only for the convenience of the description of the present application, and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0060] In related technologies, text correction is mainly based on pinyin. First, the pinyin of all local dictionaries is stored in a data structure. Then, the text to be corrected is annotated with pinyin. The edit distance between the pinyin of the text to be corrected and the pinyin of all dictionaries is calculated. By setting a certain threshold, texts with the closest pinyin edit distance to the text to be corrected are selected as candidate texts. The text with the highest score is then selected according to certain rules. For longer texts, a character-based edit distance method is used. The character-based edit distance between the input text to be corrected and the existing dictionary texts is calculated, and then the text with the highest score is calculated according to certain sorting rules. Both pinyin-based and character-based edit distance-based text correction are highly dependent on dictionaries. If the user inputs text that is not in the dictionary, the accuracy of correction is extremely low, resulting in a very poor user experience.

[0061] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a text correction method is provided.

[0062] Optionally, in the embodiments of this application, the above-described text correction method can be applied to, for example, Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1 As shown, server 103 is connected to terminal 101 via a network and can be used to provide services to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services for server 103. The network mentioned above includes, but is not limited to, wide area network, metropolitan area network or local area network. Terminal 101 includes, but is not limited to, PC, mobile phone, tablet computer, etc.

[0063] A text correction method in this application embodiment can be executed by server 103, or it can be executed jointly by server 103 and terminal 101, such as... Figure 2 As shown, the method may include the following steps:

[0064] Step S202: Obtain the text to be corrected.

[0065] In the embodiments of the present application, the text correction method provided by the present application can be deployed in technical scenarios such as question and answer systems and search engines. For example, a user inputs a piece of text in a search box and clicks search, the search result is based on the content input by the user, if the text used for searching is inconsistent with the content that the user really wants to search due to input method, human error, omission and over-input, the search result of the search engine will deviate from the original intention of the user, resulting in the user needing to re-input the text and correct the input text one by one. However, by deploying the text correction method provided by the present application, after the user inputs a piece of text, the system can determine whether the user has made a mistake, over-input or omitted input based on the text input by the user, if text correction is needed, the text correction method provided by the embodiments of the present application is used for text correction, so that the text with more accurate semantics and more in line with the original intention of the user is used as search text for content search, and the search content obtained is more in line with the user's demand, thereby improving the user experience.

[0066] In the embodiments of the present application, the text correction method provided by the present application can also be deployed in an input method. For example, during the process in which a user inputs a piece of text through an input method, the system can correct the text input by the user in real time, and can provide the user with a plurality of corrected texts for the user to select the text that is most in line with the original intention of the user.

[0067] In the embodiments of the present application, the text input by the user in a search box and the like can be used as the text to be corrected, and obtaining the text to be corrected only needs to read the content of the search box.

[0068] In step S204, the text to be corrected is identified based on the first knowledge graph to obtain a target wrong word in the text to be corrected, the target wrong word being a word in the text to be corrected that is not related to the content of the text to be corrected, and the first knowledge graph being used to record domain knowledge related to the content of the text to be corrected.

[0069] In the embodiments of the present application, the first knowledge graph described above is professional domain knowledge of a specific domain, and can also be general domain knowledge. For example, if the text to be corrected input by the user contains a neural network, the knowledge graph related to the neural network is called, and if the text to be corrected input by the user contains table tennis, the knowledge graph related to table tennis is called.

[0070] The knowledge graph records extensive and comprehensive domain knowledge of the corresponding domain, and based on the first knowledge graph, the content that does not match or is not related to the domain knowledge can be found from the text to be corrected, so as to determine the target wrong word.

[0071] In step S206, a candidate replacement word of the target wrong word is determined based on a second knowledge graph, and the second knowledge graph is used to record domain knowledge of easily confused texts.

[0072] In the embodiments of the present application, the second knowledge graph is a knowledge graph recording easily confused words, such as homophonic confusion, homonym confusion, near-homophonic confusion, near-homonym confusion, and similar words, etc. The second knowledge graph can be used to find out the replacement combination of the target wrong word, i.e., the candidate replacement word.

[0073] In step S208, the candidate text obtained by replacing the target wrong word with the candidate replacement word is subjected to feature sorting.

[0074] In the embodiments of the present application, after a candidate word replaces the target wrong word, a candidate text is obtained. After all candidate words are replaced, all candidate texts are subjected to feature sorting, so as to judge the rationality of each candidate text, whether the semantics is accurate, and whether it conforms to the domain knowledge.

[0075] In step S210, the optimal text subjected to feature sorting is determined as the corrected text.

[0076] In the embodiments of the present application, the optimal text obtained after sorting is determined as the corrected text. The optimal text is the text that is most reasonable, has the most accurate semantics, and conforms to the related domain knowledge, judged by the model.

[0077] Through the above steps S202 to S210, the present application adopts the error detection, candidate recall, and candidate sorting mode to correct the text in combination with the knowledge graph. The dictionary and confusion data set of the domain knowledge are obtained by mining the knowledge graph, and the recall of the candidate word is increased. The result of the text correction is more accurate, and the technical problem of low accuracy of text correction is solved.

[0078] Optionally, in step S204, the first knowledge graph is used to identify the target wrong word in the text to be corrected, including:

[0079] In step 1, the target dictionary is extracted from the first knowledge graph, and the text to be corrected is preprocessed. The content of the target dictionary includes the target attribute value of the corresponding entity in the first knowledge graph. The preprocessing includes at least one of format conversion, format standardization, and character filtering.

[0080] In the embodiment of the present application, the target dictionary is extracted from the knowledge graph database. First, the attributes of each entity to be obtained in the knowledge graph are determined. As an optional implementation, the present application can obtain the main and sub-title attributes of each entity (for example, the entity of company employee, the main title attribute is employee name, and the sub-title attribute is employee alias, so that the name and alias of each employee are stored in the error correction dictionary), and then obtain all values of the main and sub-title attributes of each entity. After determining the attributes of the entity to be obtained, the attribute values of the entity can be read from the distributed file system to read the knowledge graph, and then all values of the entity corresponding to the attribute can be obtained, or the interface of the knowledge graph can be called by the gremlin method to obtain all values of the entity corresponding to the attribute. Gremlin language is the most mainstream query language of graph database, and is the standard graph language under Apache TinkerPop framework. In Gremlin, the vertex can be queried by Step V() statement, the edge can be queried by Step E() statement, the id of the vertex (entity) and edge can be obtained by Step id() statement, the label of the vertex and edge can be obtained by Step label() statement, and the properties of the vertex and edge can be obtained by Step properties() statement.

[0081] In the embodiment of the present application, the target dictionary can also be obtained from the log of historical search records, the log of user's historical search is analyzed to obtain the query (query text) used by the user each time, and the historical search query of the user is stored in the dictionary.

[0082] In the embodiment of the present application, format conversion, format standardization and character filtering can remove meaningless characters, symbols and spaces in the text to be corrected, for example, converting full-width characters to half-width characters, removing useless spaces, punctuation marks and expressions, etc. The text to be corrected can be standardized to remove redundant information.

[0083] In step 2, the preprocessed text to be corrected is split into multiple character combinations using the target dictionary, and each character combination is matched with the data in the target dictionary, wherein each character is assigned to at least one character combination, and each character combination includes at least one character.

[0084] In the embodiment of the present application, the target dictionary can be used for word segmentation of the text to be corrected. Since word segmentation is based on dictionary segmentation, the segmented word is a single character which is not considered to form a word with context and is regarded as a possible wrong word. At this time, each possible wrong word can be combined with the surrounding words. If the combined word appears in the dictionary, the position is not a wrong word. Thus, the possible wrong words and wrong characters in the text to be corrected can be found.

[0085] In the embodiments of the present application, the HanLP tool can be used for word segmentation. HanLP is a Java toolkit composed of a series of models and algorithms, aiming to popularize the application of natural language processing in production environment. HanLP has the characteristics of perfect function, high performance, clear architecture, corpus time, and customizability. While providing rich functions, the internal modules of HanLP adhere to low coupling, the models adhere to lazy loading, the services adhere to static provision, and the dictionaries adhere to plaintext release, which is very convenient to use. At the same time, it comes with some corpus processing tools to help users train their own corpus.

[0086] In the embodiments of the present application, it should be noted that for the corresponding field, a custom field dictionary needs to be introduced, and the dictionary data is provided by the entity attribute value of the knowledge graph, so that the word segmenter can recognize field-specific terms and improve the accuracy of word segmentation. For example, the word segmentation result of "credit card reflects the hand fee" is "credit card / reflects / hand / fee", and "hand", "must", and "fee" are regarded as potential incorrect words.

[0087] Step 3, determine the character combination that does not match the data in the target dictionary as the target wrong word.

[0088] In the embodiments of the present application, "hand / must / fee" is completely unmatched with the domain knowledge of the knowledge graph related to credit cards, and cannot form reasonable words before and after, so "hand / must / fee" can be determined as the target wrong word.

[0089] For some professional field words with the same pronunciation but different meanings, such as "Mingcha" and "Mingcha", "Mingzhi system" and "Meiji system", etc., only word segmentation cannot find the potential wrong words, so it is necessary to detect the errors of the professional field words based on the knowledge graph.

[0090] Optionally, the target wrong word in the text to be corrected is obtained by identifying the text to be corrected based on the first knowledge graph, and the method further includes:

[0091] Step 1, creating a dictionary tree using the first knowledge graph, the dictionary tree is used to store the mapping relationship from pinyin to entity of the domain knowledge related to the content of the text to be corrected, and each node in the dictionary tree stores a pinyin character in the pinyin;

[0092] Step 2, converting the character combination into a pinyin combination, and determining a target pinyin chain with the minimum edit distance of the pinyin combination in the dictionary tree;

[0093] Step 3, determining a preset character combination of the entity corresponding to the target pinyin chain according to the mapping relationship from pinyin to entity;

[0094] Step 4, determining the character combination as the target wrong word in the case that the character combination is inconsistent with the preset character combination.

[0095] In the implementation of the present application, the professional field words such as “Mingcha” and “Mingzhi”, “Mingzhixitong” and “Mingzhixitong” are mostly homophonic errors. Therefore, in the embodiments of the present application, a dictionary tree is created by using a first knowledge graph, the dictionary tree is used to store the mapping relationship of pinyin to entity of the domain knowledge related to the content of the text to be corrected, and each node in the dictionary tree stores a pinyin character in the pinyin, thereby establishing a mapping dictionary of pinyin to entity in the professional field, and thus completing the error correction process from wrong words to pinyin and then to entity. For example, in the pinyin “mingcha” of “Mingcha”, m is taken as the root node of the tree, and the child nodes i, n, g, c, h and a are created in turn to obtain the pinyin chain m-i-n-g-c-h-a, and the preset entity word corresponding to the pinyin chain is set to “Mingcha” instead of “Mingcha”, and thus the mapping relationship of m-i-n-g-c-h-a to “Mingcha” is established.

[0096] In the embodiments of the present application, taking the character combination “Mingcha” in the text to be corrected and the preset character combination “Mingcha” in the dictionary tree as an example, “Mingcha” is converted into the pinyin combination mingcha, the target pinyin chain m-i-n-g-c-h-a is found in the dictionary tree, the preset character is determined to be “Mingcha” according to the preset mapping relationship of pinyin to entity, and it is determined that “Mingcha” is not consistent with “Mingcha”, so “Mingcha” in the text to be corrected is the target wrong word.

[0097] In the embodiments of the present application, for the pinyin of homophonic and different tones, homophonic and similar tones, and similar tones and different tones, the pinyin editing distance of the pinyin chain with the minimum pinyin editing distance can be calculated as the target pinyin chain, thereby solving the problem that the pinyin of the error fragment cannot be matched when the pinyin is not in the dictionary tree. For example, the pinyin of “Mincha” is mincha, and the pinyin chain that can be matched in the dictionary tree is mingcha with a pinyin editing distance of 1. Therefore, mingcha can be determined as the target pinyin chain for text correction. In the calculation of the above pinyin editing distance, one pinyin character g needs to be inserted from mincha to obtain mingcha, so the pinyin editing distance is 1.

[0098] In the embodiments of the present application, the implementation mode of the dictionary tree can adopt a Trie tree.

[0099] For some words that can form words but have no or little relevance to the content of the text, it is also difficult to determine them as wrong words by word segmentation. Therefore, the present technical solution also provides an error detection based on a bert model.

[0100] Optionally, determining the target wrong word in the text to be corrected further includes:

[0101] Step 1: According to the order of the characters in the text to be corrected, each character combination in the text to be corrected is covered in turn to obtain a plurality of covered text sequences;

[0102] Step 2, input the covered text sequence into the mask language model, and obtain the predicted character and the corresponding predicted probability output by the mask language model, the predicted character being a character filling the covered area recognized by the mask language model according to the context semantics of the covered character, and the predicted probability being the probability of the predicted character filling the covered area.

[0103] Step 3, in the case that the combination of the covered characters is inconsistent with the corresponding predicted character, the covered character is determined as the target wrong word.

[0104] In the embodiment of the application, the Mask Language Model (mask language model) is a bert model, which can convert the error detection process into a word-level cloze problem, that is, predict the word to be filled in each blank one by one during error detection, so as to judge whether the word at the current position is wrong according to the probability distribution of the prediction.

[0105] According to the order of the characters in the text to be corrected, each character in the text sentence is masked in the application, for example, the text to be corrected is {I am Chinese benevolent}, and a plurality of covered text sequences are obtained after covering, such as {mask, is, China, country, benevolent}, {I, mask, China, country, benevolent}... {I, is, China, country, mask}, so as to predict the most suitable word at the current position depending on the context of the current word. For example, assuming that the size of the word table is 10000, which means that a 10000 classification is performed at each position in the sentence, that is, the probability of selecting 10000 words to fill in is calculated). As an optional implementation, a fault tolerance threshold k = 10 can be set, and the top 10 words with the highest probability are selected from the above classification results such as 10000 classification. If the original word appears in the top 10 predicted results, it is considered that the position is not a wrong word (word), otherwise it is a wrong word (word). Further, if the position is determined as the target wrong word, the word with the highest probability in the above top 10 words is taken as the candidate replacement word.

[0106] It should be noted that the deep learning model can learn the context semantics, the bert model can better learn the context semantics of the sentence by relying on the pre-trained model, the bert can detect the knowledge correlation error of the sentence, and can predict the probability of each word appearing at each position in the sentence to be corrected. If the probability of the word at the position in the sentence is too small, it is judged as a potential error, which is convenient for subsequent better correction. The dependence on the dictionary is small, and the corrected result is also not in the dictionary. For example, in the above "credit card / embodiment / hand / must / fee", the word segmentation cannot determine "embodiment" as a wrong word, but the mask language model can recognize that the word is not related to the context and is a wrong word.

[0107] Optionally, as Figure 3As shown, after preprocessing the input sentence, this application can superimpose the error detection results based on word segmentation, the error detection results of professional field vocabulary based on the trie tree (i.e., product proper name detection), and the error detection results based on the masked language model (bert model) to obtain all the target misspelled words in the text to be corrected.

[0108] Optionally, step S206 for determining candidate replacement words for the target misspelled words based on the second knowledge graph includes:

[0109] Step 1, extract the confusion word dataset from the second knowledge graph;

[0110] Step 2, determine the words in the confusion word dataset whose similarity to the target misspelled word is greater than or equal to the similarity threshold as candidate replacement words.

[0111] In the embodiment of this application, the above-mentioned second knowledge graph is the domain knowledge graph of confusion words. Taking the Chinese character "情" as an example, the confusion words can include:

[0112] Same pronunciation and same tone: 檠晴擎[qing2], indicating that the pinyin is qing and the tone is the second tone;

[0113] Same pronunciation and different tones: 青轻清[qing1]顷请[qing3]庆磬[qing4], indicating that the pinyin is qing and the tones are the first tone, the third tone, and the fourth tone;

[0114] Similar pronunciation and same tone: 擒禽噙琴勤秦芹[qin2];

[0115] Similar pronunciation and different tones: 精经睛…京[jing1], 颈警…井[jing3], 竟径镜…敬[jing4], 今津…金[jin1], 仅锦紧…谨[jin3], 近尽禁…浸[jin4], 亲嵚[qin1], 寝[qin3], 揿[qin4], etc.

[0116] It can also include similar-looking characters: 清晴请青债渍啧惆悸惟惕, etc.

[0117] In addition, some legitimate words may also be confusion words, for example, "人员" ([ren2-yuan2], referring to'members'), "人缘" ([ren2-yuan2], referring to'relationships'). For the word "语言", there are the following confusion words: 寓言, 预演, 预言, 鱼眼, etc.,

[0118] In the embodiment of this application, the above-mentioned confusion word data can also use the publicly available perplexity word set.

[0119] In the embodiments of the present application, all the confused words of the target wrong word in the confused word data set can be directly found as candidate replacement words. The similarity between the target wrong word and each word in the confused word data set can be calculated to find the closest word, and then the closest word and its confused word are selected as candidate replacement words. The similarity can be calculated by the Euclidean distance, Pearson correlation coefficient, cosine distance, generalized Jaccard coefficient, etc.

[0120] In the embodiments of the present application, as shown in Figure 4 For example, "credit card" is an accurate word, and "cash" is a legal word group, but the semantics and context, especially the "credit card", are not consistent, and it is determined as a wrong word. The candidate replacement word of the word "cash" obtained by using the confused word data set is "withdrawal". According to the word segmentation, "hand / need / fee" is not a legal word group, so the candidate replacement word of "hand" obtained by using the confused word data set is "first", and the candidate replacement word of "need" is "need", "continue", etc., and the candidate replacement word of "fee" is "fly", etc.

[0121] Optionally, determining the candidate replacement word of the target wrong word further includes at least one of the following manners:

[0122] combining the preset characters to determine the candidate replacement word;

[0123] In a case where the prediction probability of the prediction character is greater than or equal to the probability threshold, the prediction character is determined as the candidate replacement word.

[0124] In the embodiments of the present application, when the dictionary tree is used for error detection, the preset character combination corresponding to the target pinyin chain can be used as the candidate replacement word. When the mask language model is used for error detection, in a case where the prediction probability of the prediction character is greater than or equal to the probability threshold, the prediction character can be used as the candidate replacement word.

[0125] Optionally, the feature sorting of the candidate text obtained by replacing the target wrong word with the candidate replacement word includes:

[0126] inputting the candidate text into a logistic regression model, and obtaining a text sorting result output by the logistic regression model, the logistic regression model being used for extracting text features and performing feature sorting, and the text sorting result being a similarity sorting result of the text features of the candidate text and the content features of the text to be corrected;

[0127] The text features include at least one of the following: the selection frequency of the candidate text; the edit distance between the candidate text and the text to be corrected; the Jaccard distance between the pinyin of the candidate text and the pinyin of the text to be corrected; and the semantic accuracy of the candidate text determined by a multilingual model.

[0128] In the embodiments of the present application, as shown in Figure 4As shown, the candidate texts include "credit card cash hand fee", "credit card cash hand fee", "credit card cash hand fee", "credit card cash hand fee", "credit card cash hand fee", etc. The logistic regression model is a regression analysis of independent variables and dependent variables, that is, to describe the relationship between independent variables X and dependent variables Y, or the degree of influence of independent variables X on dependent variables Y, and to predict dependent variables Y. The dependent variable is the result we want to obtain, the independent variable is the potential factor that affects the result, and the independent variable can be one or more. Among them, the independent variable is the above text features of the candidate text, and the dependent variable is the ranking result of the candidate text.

[0129] In the embodiment of the application, as Figure 5 shown, each feature can be scored, and the scores of each feature are combined for ranking, so that the candidate text with the highest score can be selected as the final text, that is, the corrected text. Among the above text features, the higher the score of the candidate text, the higher the score of the candidate text, the smaller the edit distance between the candidate text and the text to be corrected, the more similar the Jaccard distance between the pinyin of the candidate text and the pinyin of the text to be corrected, and the higher the semantic accuracy of the candidate text determined by the multi-language model. The multi-language model (n-gram) can evaluate whether the sentence is reasonable.

[0130] According to another aspect of the embodiment of the application, as Figure 6 shown, a text correction device is provided, comprising:

[0131] The text acquisition module 601 is configured to acquire a text to be corrected.

[0132] The error detection module 603 is configured to identify the text to be corrected based on a first knowledge graph to obtain a target error word in the text to be corrected, the target error word being a word in the text to be corrected that is not related to the content of the text to be corrected, and the first knowledge graph being used to record domain knowledge related to the content of the text to be corrected.

[0133] The candidate recall module 605 is configured to determine a candidate replacement word of the target error word based on a second knowledge graph, and the second knowledge graph is used to record domain knowledge of easily confused texts.

[0134] The candidate ranking module 607 is configured to rank candidate texts obtained by replacing the target error word with the candidate replacement word based on features.

[0135] The error correction module 609 is configured to determine the optimal text ranked based on the features as a corrected text.

[0136] It should be noted that the text acquisition module 601 in this embodiment can be used to execute step S202 in the embodiments of the present application, the error detection module 603 in this embodiment can be used to execute step S204 in the embodiments of the present application, the candidate recall module 605 in this embodiment can be used to execute step S206 in the embodiments of the present application, the candidate sorting module 607 in this embodiment can be used to execute step S208 in the embodiments of the present application, and the error correction module 609 in this embodiment can be used to execute step S210 in the embodiments of the present application.

[0137] It should be noted here that the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can run in the hardware environment as shown in Figure 1 It can be implemented by software or by hardware.

[0138] Optionally, the error detection module is specifically configured to:

[0139] extract a target dictionary from the first knowledge graph, and pre-process the text to be corrected, the content of the target dictionary including target attribute values of corresponding entities in the first knowledge graph, the pre-processing including at least one of format conversion, format standardization, and character filtering;

[0140] split the pre-processed text to be corrected into a plurality of character combinations using the target dictionary, and match each character combination with data in the target dictionary, wherein each character is assigned to at least one character combination, and each character combination includes at least one character;

[0141] determine a target wrong word as a character combination that does not match the data in the target dictionary.

[0142] Optionally, the error detection module is further configured to:

[0143] create a dictionary tree using the first knowledge graph, the dictionary tree being used to store a pinyin-to-entity mapping relationship of domain knowledge related to the content of the text to be corrected, and each node in the dictionary tree corresponding to storing one pinyin character in the pinyin;

[0144] convert the character combination into a pinyin combination, and determine a target pinyin chain with the smallest edit distance to the pinyin combination in the dictionary tree;

[0145] determine a preset character combination of an entity corresponding to the target pinyin chain according to the pinyin-to-entity mapping relationship;

[0146] in the case where the character combination is inconsistent with the preset character combination, determine the character combination as a target wrong word.

[0147] Optionally, the error detection module is further configured to:

[0148] The error detection module is further configured to:

[0149] The error detection module is further configured to:

[0150] The error detection module is further configured to:

[0151] Optionally, the candidate recall module is specifically configured to:

[0152] The candidate recall module is specifically configured to:

[0153] The candidate recall module is specifically configured to:

[0154] Optionally, the candidate recall module is further configured to:

[0155] The candidate recall module is further configured to:

[0156] The candidate recall module is further configured to:

[0157] Optionally, the candidate sorting module is specifically configured to:

[0158] The candidate sorting module is specifically configured to:

[0159] The text features include at least one of:

[0160] The selection frequency of the candidate text;

[0161] The edit distance between the candidate text and the text to be corrected;

[0162] The Jaccard distance between the pinyin of the candidate text and the pinyin of the text to be corrected;

[0163] The semantic accuracy of the candidate text determined by the multilingual language model.

[0164] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 7 As shown in FIG. 7, the electronic device includes a memory 701, a processor 703, a communication interface 705, and a communication bus 707. The memory 701 stores a computer program executable on the processor 703. The memory 701 and the processor 703 communicate through the communication interface 705 and the communication bus 707. The processor 703 executes the computer program to implement the steps of the above method.

[0165] The memory and the processor in the electronic device communicate through the communication bus and the communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0166] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0167] The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. The processor can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0168] According to yet another aspect of the embodiments of the present application, a computer readable medium having non-volatile program code executable by a processor is also provided.

[0169] Optionally, in the embodiments of the present application, the computer readable medium is configured to store program code for the processor to execute the following steps:

[0170] Obtaining the text to be corrected;

[0171] The first knowledge graph is used for recording domain knowledge related to the content of the to-be-corrected text.

[0172] The second knowledge graph is used for recording domain knowledge of the easily-confused text.

[0173] The candidate text obtained by replacing the target wrong word with the candidate replacement word is subjected to feature sorting.

[0174] The optimal text subjected to feature sorting is determined as the corrected text.

[0175] Optionally, specific examples in the embodiments can refer to the examples described in the above embodiments, and the embodiments will not be described here again.

[0176] In the implementation, the embodiments can refer to the above embodiments and have corresponding technical effects.

[0177] It can be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described in the embodiments, or a combination thereof.

[0178] For software implementation, the technologies described herein can be implemented by units performing the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0179] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0180] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0181] In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between the devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0183] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0184] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, and various program codes that can be stored in the medium. It should be noted that, in this document, relationship terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0185] The above is only a specific embodiment of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications of these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A method of text correction, characterized by, The method comprises: obtaining a text to be corrected; identifying the text to be corrected based on a first knowledge graph to obtain a target wrong word in the text to be corrected, wherein the target wrong word is a word in the text to be corrected that is not related to the content of the text to be corrected, and the first knowledge graph is used to record domain knowledge related to the content of the text to be corrected; determining a candidate replacement word for the target wrong word based on a second knowledge graph, wherein the second knowledge graph is used to record domain knowledge of confusing text; performing feature sorting on a candidate text obtained by replacing the target wrong word with the candidate replacement word; determining the optimal text of the feature sorting as a corrected text; The method further comprises: superimposing an error detection result based on word segmentation matching, an error detection result based on a dictionary tree professional field vocabulary, and an error detection result based on a mask language model to obtain all target wrong words in the text to be corrected.

2. The method of claim 1, wherein, The method further comprises: extracting a target dictionary from the first knowledge graph and preprocessing the text to be corrected, wherein the content of the target dictionary includes target attribute values of corresponding entities in the first knowledge graph, and the preprocessing includes at least one of format conversion, format standardization, and character filtering; splitting the preprocessed text to be corrected into a plurality of character combinations using the target dictionary, and matching each character combination with data in the target dictionary, wherein each character is assigned to at least one character combination, and each character combination includes at least one character; determining the character combination that does not match the data in the target dictionary as the target wrong word.

3. The method of claim 2, wherein, The method further comprises: creating a dictionary tree using the first knowledge graph, wherein the dictionary tree is used to store a pinyin-to-entity mapping relationship of domain knowledge related to the content of the text to be corrected, and each node in the dictionary tree stores a pinyin character in the pinyin; converting the character combination into a pinyin combination and determining a target pinyin chain with the smallest edit distance to the pinyin combination in the dictionary tree; determining a preset character combination of an entity corresponding to the target pinyin chain according to the pinyin-to-entity mapping relationship; in the case where the character combination is inconsistent with the preset character combination, determining the character combination as the target wrong word.

4. The method of claim 3, wherein, The method further comprises: sequentially covering each character combination in the text to be corrected according to the order of the text in the text to be corrected to obtain a plurality of covered text sequences; input the covered text sequence into a mask language model, and obtain predicted characters and corresponding prediction probabilities output by the mask language model, wherein the predicted characters are characters filling the covered areas identified by the mask language model according to the context semantics of the covered characters, and the prediction probabilities are probabilities of the predicted characters filling the covered areas; in a case where the combination of the covered characters is inconsistent with the corresponding predicted characters, the covered characters are determined as the target wrong word.

5. The method of claim 4, wherein, determining candidate replacement words of the target wrong word based on a second knowledge graph includes: extracting a confused word data set from the second knowledge graph; determining, from the confused word data set, a word having a similarity greater than or equal to a similarity threshold to the target wrong word as the candidate replacement word.

6. The method according to claim 4 or 5, characterized in that, determining the candidate replacement words of the target wrong word further includes at least one of the following manners: determining the preset character combination as the candidate replacement word; in a case where the prediction probability of the predicted character is greater than or equal to a probability threshold, determining the predicted character as the candidate replacement word.

7. The method of claim 6, wherein, performing feature sorting on a candidate text obtained by replacing the target wrong word with the candidate replacement word includes: inputting the candidate text into a logistic regression model, and obtaining a text sorting result output by the logistic regression model, wherein the logistic regression model is used to extract text features and perform feature sorting, and the text sorting result is a similarity sorting result of text features of the candidate text and content features of the text to be corrected; the text features include at least one of the following: a selection frequency of the candidate text; an edit distance between the candidate text and the text to be corrected; a Jaccard distance between pinyin of the candidate text and pinyin of the text to be corrected; a semantic accuracy of the candidate text determined by a multilingual model.

8. A text correction apparatus characterized by comprising: includes: a text acquisition module configured to acquire a text to be corrected; an error detection module configured to identify the text to be corrected based on a first knowledge graph to obtain a target wrong word in the text to be corrected, wherein the target wrong word is a word irrelevant to content of the text to be corrected, and the first knowledge graph is used to record domain knowledge related to the content of the text to be corrected; a candidate recall module configured to determine candidate replacement words of the target wrong word based on a second knowledge graph, wherein the second knowledge graph is used to record domain knowledge of confused texts; a candidate sorting module configured to perform feature sorting on a candidate text obtained by replacing the target wrong word with the candidate replacement word; a correction module configured to determine an optimal text of the feature sorting as a corrected text; the error detection module is further configured to superimpose an error detection result based on word segmentation matching, an error detection result based on a dictionary tree of professional domain words, and an error detection result based on a mask language model to obtain all the target wrong words in the text to be corrected.

9. An electronic device comprising a memory, a processor, a communication interface and a communication bus, the memory storing a computer program executable on the processor, the memory, the processor communicating through the communication bus and the communication interface, characterized in that, the processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A computer readable medium having non-transitory program code executable by a processor, the program code comprising instructions for: the program code causes the processor to execute the method of any one of claims 1 to 7.

Images