Text error correction method and related equipment

Abstract

The present application provides a text error correction method and related equipment, including: obtaining a text to be corrected; segmenting the text to be corrected by a word segmentation algorithm to obtain a word segmentation error correction text; performing error detection on the text to be corrected and the word segmentation error correction text respectively to obtain an error candidate set, wherein the error candidate set includes error words; determining a replacement candidate set based on the error candidate set; grouping the error candidate set to obtain multiple error candidate subsets; generating multiple candidate paths based on multiple error candidate subsets; determining a candidate recommendation set based on the replacement candidate set and the candidate path; and obtaining a recommended text based on the recommended candidate set by a preset algorithm. The text error correction method provided by the present application can cover all error correction paths and candidate possibilities, thereby improving the accuracy of text error correction.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text error correction method and related equipment. Background Art

[0002] Text error correction technology is a technology that identifies spelling errors and their location information in input text and gives correct modification suggestions. Text error correction is a very important task in the field of natural language processing. It has a wide range of applications in search engines, speech recognition and other fields. It is generally used in the text preprocessing stage and can significantly improve the accuracy of semantics in various scenarios.

[0003] Existing error correction methods generally include: error detection, candidate recall, and candidate sorting. The error detection step needs to detect the position of the erroneous character or word in the text, and the candidate recall step needs to traverse all suspected error positions and give corresponding replacement candidate character or word sets. Finally, the candidate characters or words in the recalled candidate set need to be sorted, the unique candidate is taken out, and the erroneous character or word is replaced. At present, the error detection effect of text using a character-granularity detection model is not ideal, because the character-granularity detection model cannot utilize word-level information, which makes the corrected text difficult to interpret. If the word-granularity detection model is integrated, the text needs to be segmented. However, the current fusion of the character-granularity detection model and the word-granularity detection model cannot completely solve the problem caused by the word segmentation error, resulting in a low accuracy rate of text error correction. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a text error correction method and related equipment.

[0005] Based on the above purpose, the present application provides a text error correction method, comprising:

[0006] Get the text to be corrected;

[0007] Segment the text to be corrected by a word segmentation algorithm to obtain a word segmentation correction text;

[0008] Performing error detection on the text to be corrected and the word segmentation correction text respectively to obtain an error candidate set, wherein the error candidate set includes error words;

[0009] determining a replacement candidate set based on the error candidate set;

[0010] Grouping the error candidate set to obtain multiple error candidate subsets;

[0011] generating a plurality of candidate paths based on a plurality of said error candidate subsets;

[0012] Determine a candidate recommendation set based on the replacement candidate set and the candidate paths;

[0013] The recommended text is obtained based on the recommendation candidate set through a preset algorithm.

[0014] Furthermore, the word segmentation of the text to be corrected by using a word segmentation algorithm includes:

[0015] The text to be corrected is segmented by Jieba word segmentation algorithm and CoreNLP word segmentation algorithm respectively.

[0016] Furthermore, the error detection is performed on the text to be corrected and the word segmentation correction text respectively to obtain an error candidate set, including:

[0017] Matching each character or word in the segmentation error correction text with a preset common word dictionary to obtain a first error set;

[0018] Combining any two adjacent characters or words in the word segmentation error correction text in pairs to obtain a combination word set, searching the pinyin of each combination word in the combination word set in the common word dictionary to obtain a homophone set, and determining a second error set based on the homophone set;

[0019] Outputting a predicted word set through a word granularity mask language model based on the text to be corrected, and determining a third error set based on the predicted word set;

[0020] Matching each word in the text to be corrected with a preset confusion dictionary to obtain a fourth error set;

[0021] The first error set, the second error set, the third error set, and the fourth error set are combined into the error candidate set.

[0022] Further, the determining of a replacement candidate set based on the error candidate set includes:

[0023] Each character in each word in the error candidate set is replaced with all the characters in the common word dictionary in sequence to obtain a single-character replacement set,

[0024] All characters in each word in the error candidate set are arranged and combined to obtain a rotation word set, and each character in each word in the rotation word set is replaced with all characters in the common word dictionary in sequence to obtain a rotation order set.

[0025] Matching each word in the single-word replacement set and the rotation order set with the common word dictionary to obtain a first replacement set;

[0026] Matching each character in the error candidate set with a preset dictionary of similar characters to obtain a second replacement set;

[0027] Matching each character in the error candidate set with a preset dictionary of similar-sounding characters to obtain a third replacement set;

[0028] The correct word group corresponding to each word in the fourth error set in the confusion dictionary is formed into a fourth replacement set;

[0029] The first replacement set, the second replacement set, the third replacement set, the homonymous word set, the predicted word set and the fourth replacement set are combined into the replacement candidate set.

[0030] Furthermore, before determining the candidate recommendation set based on the replacement candidate set and the candidate paths, the method further includes:

[0031] The words or phrases in the replacement candidate set whose sentence perplexity corresponding to the words or phrases exceeds a preset threshold are removed from the replacement candidate set.

[0032] Further, the generating a plurality of candidate paths based on the plurality of error candidate subsets includes:

[0033] All characters or words in the error candidate subsets are arranged and combined to obtain a plurality of candidate paths.

[0034] Further, the determining of the candidate recommendation set based on the replacement candidate set and the candidate paths includes:

[0035] The replacement candidate characters or words corresponding to each character or word in the candidate path in the replacement candidate set are combined into a candidate recommendation set corresponding to the candidate path.

[0036] Furthermore, obtaining the recommended text based on the recommendation candidate set by using a preset algorithm includes:

[0037] The recommended text is obtained by respectively calculating the recommendation candidate set using a sentence perplexity algorithm and a word granularity mask language model.

[0038] Based on the same inventive concept, the present application also provides a text error correction device, comprising:

[0039] A text acquisition module is configured to acquire the text to be corrected;

[0040] A word segmentation module is configured to segment the text to be corrected by a word segmentation algorithm to obtain a word segmentation correction text;

[0041] An error detection module is configured to perform error detection on the text to be corrected and the word segmentation correction text respectively to obtain an error candidate set;

[0042] A replacement candidate word determination module, configured to determine a replacement candidate set based on the error candidate set;

[0043] a grouping module, configured to group the error candidate set to obtain a plurality of error candidate subsets;

[0044] A path determination module, configured to generate a plurality of candidate paths based on a plurality of the error candidate subsets;

[0045] a recommendation set determination module, configured to determine a candidate recommendation set based on the replacement candidate set and the candidate paths;

[0046] The recommended text determination module is configured to obtain the recommended text through a preset algorithm based on the recommendation candidate set.

[0047] Based on the same inventive concept, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.

[0048] Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method as described above.

[0049] From the above, it can be seen that the text error correction method and related equipment provided by the present application, in view of the impact of word segmentation errors in text error correction, segment the text to be corrected through different word segmentation models, and use multiple methods to detect errors in the text to be corrected after word segmentation. Through the method of combining paths, all error correction schemes are formed into candidate error paths, and then all error combinations are obtained on a given error path to form all error correction schemes under the error path, and the error correction schemes are merged, and finally all candidate characters or words corresponding to the error are traversed at the error position in each scheme to obtain all candidate recalls. The text error correction method provided by the present application can cover all error correction paths and candidate possibilities, achieve better error correction effects, effectively reduce the problem of error position overlap caused by different word segmentation methods and segmentation methods with different character and word granularity, and can also reduce the error accumulation problem caused by correcting sentences from front to back, and improve the accuracy of text error correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic diagram of a text error correction method according to an embodiment of the present application;

[0052] Figure 2a A schematic diagram of the selection of the position of the error candidate set in an embodiment of the present application;

[0053] Figure 2b A schematic diagram of the completion of position cutting of the error candidate set in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of a text error correction device according to an embodiment of the present application;

[0055] Figure 4 This is a schematic diagram of the electronic structure hardware structure of an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0057] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0058] As described in the background technology, an important issue in the error correction task is whether to perform word segmentation. Since the Chinese word segmentation systems are trained on clean corpus, if there are typos in the sentences, word segmentation of the wrong sentences will lead to many incorrect segmentations. The wrong word segmentation results will cause the original complete words to be segmented into more fine-grained single words, or segmentation combination errors, making the errors more difficult to handle, such as "and|also|have|to|reorganize" and "My|family|letter|is|expensive|and|is|down." The prior art solves the above problems by fusing the detection models of character granularity and word granularity, but the effect is not obvious. In order to address the limitations of the prior art, the present application proposes a text error correction method, which can cover all error correction paths and candidate possibilities, alleviate the impact of word segmentation errors to a certain extent, and improve the error correction effect.

[0059] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0060] This application provides a text error correction method, reference Figure 1 , including the following steps:

[0061] Step S101, obtaining a text to be corrected. Chinese text often contains various errors, such as errors in similar-looking characters, similar-sounding characters, terminology errors, semantic errors, and errors in idioms or two-part allegorical sayings, etc. The method in this embodiment mainly corrects errors in similar-sounding characters.

[0062] Step S102: segment the text to be corrected by a word segmentation algorithm to obtain a segmented and corrected text.

[0063] Specifically, by segmenting the text to be corrected, character granularity and word granularity information of the text to be corrected are obtained, so as to perform further error correction processing on the segmented text to be corrected.

[0064] Step S103, respectively perform error detection on the text to be corrected and the segmented text to obtain an error candidate set, wherein the error candidate set includes erroneous words. The text to be corrected and the segmented text to be corrected are detected respectively by different error detection methods. The text to be corrected is detected on the basis of the original text without the segmentation process, which is error detection based on the word granularity. The segmented text to be corrected is detected on the basis of the segmentation process, which is error detection based on the included word granularity. The error candidate set finally obtained includes both erroneous words and characters.

[0065] Step S104, determining a replacement candidate set based on the error candidate set. After obtaining the error candidate set of the text to be corrected, a replacement candidate set is determined by a certain screening method, that is, a corresponding replacement candidate set is determined according to the error candidate set obtained by each different type of detection method.

[0066] Step S105: Group the error candidate set to obtain multiple error candidate subsets. In view of the problem of repeated error detection in the error candidate set obtained by multiple error detection methods, different detection methods may obtain the same text position error. By grouping the error candidate set to reduce repeated text position error detection, repeated calculations in the subsequent candidate solution generation process are avoided as much as possible, effectively improving the error correction efficiency and saving text error correction time.

[0067] Step S106: Generate multiple candidate paths based on the multiple error candidate subsets. According to different replacement candidate characters or words in the error candidate subsets, multiple candidate paths are constructed to achieve full coverage of candidate answers and avoid missing candidate answers.

[0068] Step S107: Determine a candidate recommendation set based on the candidate replacement set and the candidate paths. After the candidate paths are determined, determine a candidate recommendation set by combining the candidate replacement set obtained in step S104.

[0069] Step S108: obtaining a recommended text based on the recommended candidate set by a preset algorithm, and further screening the recommended candidate set to determine the recommended text, that is, obtaining the correct text for the text to be corrected.

[0070] In some embodiments, the word segmentation of the text to be corrected by the word segmentation algorithm includes: respectively using the Jieba word segmentation algorithm and the CoreNLP word segmentation algorithm to segment the text to be corrected. Specifically, the text to be corrected is first divided into short sentences, and then the short sentences are segmented to obtain a word segmentation sequence containing several words. In this embodiment, the Jieba word segmentation algorithm and the CoreNLP word segmentation algorithm are respectively used to segment the text to be corrected, wherein the Jieba word segmentation algorithm tends to split the words containing wrong characters into more fine-grained words, for example, "and also need to be reorganized" is split into "and|also|need|from|reorganized". The CoreNLP word segmentation algorithm tends to treat the words containing wrong characters as a whole word, for example, "and also need to be reorganized" is split into "and|also|need|from|reorganized". The word segmentation correction text obtained by the above two word segmentation algorithms is subjected to the next step of error detection.

[0071] In some embodiments, the performing error detection on the text to be corrected and the word segmentation correction text respectively to obtain an error candidate set includes:

[0072] Matching each character or word in the segmentation error correction text with a preset common word dictionary to obtain a first error set;

[0073] Combining any two adjacent characters or words in the word segmentation error correction text in pairs to obtain a combination word set, searching the pinyin of each combination word in the combination word set in the common word dictionary to obtain a homophone set, and determining a second error set based on the homophone set;

[0074] Outputting a predicted word set through a word granularity mask language model based on the text to be corrected, and determining a third error set based on the predicted word set;

[0075] Matching each word in the text to be corrected with a preset confusion dictionary to obtain a fourth error set;

[0076] The first error set, the second error set, the third error set, and the fourth error set are combined into the error candidate set.

[0077] Specifically, in this embodiment, the error detection of the text to be corrected includes four detection methods, namely:

[0078] 1) Error detection based on dictionary matching: This method traverses each character or word in the segmented error correction text after word segmentation. If the character or word is not in the preset common word dictionary, it is considered to be an error candidate character or word, and the starting position of the character or word in the sentence is output to obtain the first error set. For the characters in the segmented error correction text, they can all be matched in the common word dictionary, and for the words in the segmented error correction text, they may not be matched in the common word dictionary. Therefore, the error detection method based on dictionary matching can detect the erroneous words in the segmented error correction text.

[0079] 2) Error Detection Based on Compound Word Matching: This method is based on the segmented and error-corrected text. Two adjacent characters or words in the segmented and error-corrected text are combined pairwise to obtain a set of compound words. For example, for the segmented and error-corrected text "which|area|committee|Zhang|relatively|many", the set of compound words after combination is [which area, area committee, committee Zhang, Zhang relatively, relatively many]. Each compound word in this set of compound words is traversed to obtain the pinyin of the compound word, and a search is performed in the common word dictionary to obtain the set of homophonic words corresponding to this word. Here, homophonic words only refer to those with the same pinyin, and the tones do not need to be the same. For example, after searching, the set of homophonic words corresponding to the compound word "committee Zhang" is [slightly open, violate regulations, not increase, curtain, slightly increase]. If the set of homophonic words is empty, it is considered that there is no error in this compound word, that is, the combination method of this compound word is incorrect, and the two characters or words that make up this compound word are themselves not in error. If the set of homophonic words is not empty, first judge whether this compound word appears in the set of homophonic words. If the set of homophonic words contains this compound word, it is considered that there is no error in this compound word. If this compound word is not included in the set of homophonic words, the two characters or words that make up this compound word are used as candidate error characters or words, and the positions of these two characters or words in the text to be error-corrected are output to obtain the second error set. For the compound word "committee Zhang" in the above example, it does not appear in its set of homophonic words, indicating that "committee Zhang" is not included in the common word dictionary and does not belong to the words in normal sentences, so it is a candidate error word. Therefore, the two error candidate characters "committee" and "Zhang" and their corresponding positions in the sentence will be output. The error detection method based on compound words can effectively reduce the impact caused by segmentation errors, effectively avoid the incorrect understanding of the semantics of the text to be error-corrected due to the incorrect segmentation of characters or words by the segmentation algorithm, and improve the text error correction accuracy rate.

[0080] 3) Error Detection Based on Character-Granularity Masked Language Model: This method is used to detect the text to be error-corrected without segmenting it. Based on the text to be error-corrected, each character is traversed, and the Mask masking method in the pre-trained language model Bert-Chinese is used to mask this character, and predictions are made on the masked positions. The top K predicted characters are selected to form a set of predicted characters, where K is an integer greater than zero. If this character is not in this set of predicted characters, it is considered that this character is a candidate wrong character, and this character or word and its starting position in the sentence are output to obtain the third error set.

[0081] 4) Error detection based on confusion dictionary: The confusion dictionary is a predefined dictionary that collects common easily confused words in the existing corpus. The word format in the confusion dictionary is "wrong word-correct word", "wrong word" as the key, and "correct value" as the value. The corresponding value can be queried through the key. The wrong word and the correct word are a one-to-one mapping relationship, such as "let sit"-"let sit". First, sort all the wrong words in the confusion dictionary according to their length, and the length of the wrong word starts from 2. Then, the text sentence to be corrected is divided according to the length of the wrong word in turn, and each word obtained by the division is compared with each word in the confusion dictionary. If the word appears in the confusion dictionary, it is considered to be a candidate wrong word, and the word and its position in the text to be corrected are output to obtain the fourth error set.

[0082] The first error set, the second error set, the third error set and the fourth error set obtained by the above four methods are combined into the error candidate set.

[0083] In some embodiments, determining a replacement candidate set based on the error candidate set comprises:

[0084] Each character in each word in the error candidate set is replaced with all the characters in the common word dictionary in sequence to obtain a single-character replacement set,

[0085] All characters in each word in the error candidate set are arranged and combined to obtain a rotation word set, and each character in each word in the rotation word set is replaced with all characters in the common word dictionary in sequence to obtain a rotation order set.

[0086] Matching each word in the single-word replacement set and the rotation order set with the common word dictionary to obtain a first replacement set;

[0087] Matching each character in the error candidate set with a preset dictionary of similar characters to obtain a second replacement set;

[0088] Matching each character in the error candidate set with a preset dictionary of similar-sounding characters to obtain a third replacement set;

[0089] The correct word group corresponding to each word in the fourth error set in the confusion dictionary is formed into a fourth replacement set;

[0090] The first replacement set, the second replacement set, the third replacement set, the homonymous word set, the predicted word set and the fourth replacement set are combined into the replacement candidate set.

[0091] Specifically, the recall of the replacement candidate set includes the following six methods:

[0092] 1) Recall method based on substitution and rotation: For the misspelled words in the error candidate set, each character in the misspelled word is successively substituted with all the characters in the common word dictionary. For example, for the misspelled word "从新", substitute "从" and "新" respectively, and combine the substituted words into a single-character substitution set. Then, rotate the order of all the characters in the misspelled word to obtain a rotated word set, and for each character in each rotated word in the rotated word set, successively substitute it with all the characters in the common word dictionary to obtain a rotation order set. For example, rotate the misspelled word "从新" to get "新从", and substitute "新" and "从" respectively for single-character substitution, and the substitution results are combined into a rotation order set. Match each word in the single-character substitution set and the rotation order set with the common word dictionary respectively, and output the words that are the same as those in the common word dictionary as the first substitution set.

[0093] 2) Recall method based on similar-shaped characters: Traverse each character in the error candidate set, and find all the similar-shaped characters of this character in the preset similar-shaped character dictionary as the replacement characters of this character. For the misspelled words in the error candidate set, output the words that have undergone similar-shaped character substitution and appear in the common word dictionary as the replacement words of this word. For the misspelled characters in the error candidate set, directly output all the similar-shaped characters of this character as the replacement characters of this character. Combine all the replacement words and the replacement characters to obtain a second substitution set.

[0094] 3) Recall method based on similar-sounding characters: Similar to the recall method based on similar-shaped characters, traverse each character in the error candidate set, and find all the similar-sounding characters of this character in the preset similar-sounding character dictionary as the replacement characters of this character. For the misspelled words in the error candidate set, output the words that have undergone similar-sounding character substitution and appear in the common word dictionary as the replacement words of this word. For the misspelled characters in the error candidate set, directly output all the similar-sounding characters of this character as the replacement characters of this character. Combine all the replacement words and the replacement characters to obtain a third substitution set.

[0095] 4) Recall method based on homophone set: This method is aimed at the second error set obtained by the error detection method based on combined word matching in the error detection step. For the detected error combined word, obtain the pinyin of this combined word, and search in the common word dictionary according to the pinyin for all the word sets with this pinyin as the candidate replacement words of this combined word, that is, take the homophone set as the candidate replacement words of this combined word.

[0096] 5) Recall method based on masked language model: For each word in the error candidate set, the Mask method in the Bert-Chinese pre-trained language model is used to mask the word in the sentence, and the position is predicted. The first K words in the prediction result are output as candidate replacement words, that is, the predicted word set is output as candidate replacement words. The value of K in this method can be slightly larger than the value of K in the detection method based on the masked language model, in order to maximize the coverage of replacement words and reduce the omission of replacement words.

[0097] 6) Recall method based on confusion dictionary: This method recalls candidate answers for the error detection method based on the confusion dictionary, and outputs the correct word corresponding to each word in the confusion dictionary in the fourth error set as the fourth replacement set.

[0098] The first replacement set, the second replacement set, the third replacement set, the homophone set, the predicted word set and the fourth replacement set obtained by the six recall methods are combined into the replacement candidate set.

[0099] In some embodiments, before determining the candidate recommendation set based on the replacement candidate set and the candidate path, the method further includes:

[0100] The words or phrases in the replacement candidate set whose sentence perplexity corresponding to the words or phrases exceeds a preset threshold are removed from the replacement candidate set.

[0101] Specifically, the obtained replacement candidate set is roughly screened, and the error candidate set is determined according to different error detection methods. The replacement candidate set is determined according to the error candidate set through six candidate answer recall methods. At this time, the replacement candidate set includes a large number of replacement candidate characters or words. These replacement characters or words are replaced in the text to be corrected to calculate the sentence perplexity, and the replacement characters or words corresponding to the sentences whose sentence perplexity exceeds a preset threshold are eliminated. The rough screening process in this embodiment can filter out some replacement candidate characters or words that lead to low sentence fluency, avoid further screening and calculation, save text correction time, and effectively improve text correction efficiency.

[0102] When setting the preset threshold, the specific threshold value can be determined according to different candidate answer recall methods. For example, the preset thresholds corresponding to different recall methods can be set to different M values, and the first M candidate words in each recall method are output as the replacement candidate characters or words returned by the recall method, where M is an integer greater than zero.

[0103] In some embodiments, the error candidate set is grouped to obtain a plurality of error candidate subsets.

[0104] Specifically, since different error detection methods may produce overlapping error correction positions. For example, for the text to be error-corrected "你生体怎码样" (How are you physically), the set of error candidates obtained is [生, 体, 生体, 码, 样, 怎码样]. To obtain non-overlapping paths, the set of error candidates needs to be cut and grouped. The position cutting principle is that the character at the cutting position is the last character of all its shared candidate words. As Figure 2a shown, taking the two characters "体" and "码" as an example, the shared candidates of the character "码" in the figure are "怎码样" and "码", but the character "码" is not the last character of "怎码样". Therefore, the position where the character "码" is located does not conform to the position cutting principle. The shared candidates of "体" in the figure are "体" and "生体". Since the character "体" is the last character of both "体" and "生体" at the same time, the position where the character "体" is located conforms to the position cutting principle. The candidate set after cutting is as Figure 2b shown. According to the result of position cutting, we divide the original set of error candidates [生, 体, 生体, 码, 样, 怎码样] into two subsets of error candidates. One subset of error candidates is [生, 体, 生体], and the other subset of error candidates is [怎码样, 码, 样].

[0105] In some embodiments, generating a plurality of candidate paths based on the plurality of subsets of error candidates includes:

[0106] Performing permutations and combinations on all the characters or words in all the subsets of error candidates to obtain the plurality of candidate paths.

[0107] Taking the two subsets of error candidates [生, 体, 生体] and [怎码样, 码, 样] obtained in the above embodiments as an example to illustrate the method for generating candidate paths. Performing permutations and combinations on each character or word in the subset of error candidates [生, 体, 生体] with the characters or words in the subset of error candidates [怎码样, 码, 样], the following plurality of candidate paths can be obtained:

[0108] Path 1: [生, 怎码样]

[0109] Path 2: [生, 码]

[0110] Path 3: [生, 样]

[0111] Path 4: [生, 码, 样]

[0112] Path 5: [体, 怎码样]

[0113] Path 6: [体, 码]

[0114] Path 7: [体, 样]

[0115] Path 8: [体, 码, 样]

[0116] Path 9: [生体, 怎码样]

[0117] Path 10: [Biological entity, code]

[0118] Path 11: [Biological entity, sample]

[0119] Path 12: [Biological entity, code, sample].

[0120] The method for generating the above candidate paths includes all possible path combinations, overcoming the error propagation error in the existing solutions where the sentence errors that appear later in the text to be corrected need to be corrected based on the previously modified content, and the problem of omission of the actual correct modification paths in the subsequent error correction process due to the selection of the previously modified content in the text to be corrected.

[0121] In some embodiments, determining the candidate recommendation set based on the replacement candidate set and the candidate paths includes:

[0122] Combining the replacement candidate words corresponding to each word or term in the candidate path in the replacement candidate set into the candidate recommendation set corresponding to the candidate path.

[0123] Specifically, traverse all the paths obtained above. For a given path, such as Path 1: [How, code, sample], the set of replacement candidate words corresponding to "How" in the replacement candidate set is [Body, How, Rise], and the set of replacement candidate words corresponding to "code, sample" in the replacement candidate set is [How about, code, sample]. The final candidate recommendation set for Path 1 is {[Body, How about], [Body, code, sample], [How, How about], [How, code, sample], [Rise, How about], [Rise, code, sample]}. Perform the above operation for each of the above paths, and finally merge the candidate recommendation sets corresponding to each path to obtain all the candidate recommendation sets for all paths.

[0124] In some embodiments, obtaining the recommended text based on the candidate recommendation set through a preset algorithm includes:

[0125] Calculating the recommended text by using the sentence perplexity algorithm and the word-level masked language model on the candidate recommendation set respectively.

[0126] Specifically, in this embodiment, the candidate recommendation set is screened in a manner that combines the sentence perplexity score and the sentence smoothness score obtained based on the word-level masked language model, and finally the recommended text corresponding to the text to be corrected is obtained. The specific screening method includes the following steps:

[0127] Step S201: Calculate the sentence perplexity (ppl, perplexity) score for each sentence corresponding to each recommended candidate word or term in the candidate recommendation set. The formula for calculating the sentence perplexity ppl is:

[0128]

[0129] Among them, S represents the sentence, |S| represents the length of the sentence, and S ＜t =(w 1 , ..., w t-1 ), w i Represents each word in the sentence, 1≤i≤t, P LM (w t |S ＜t ) represents the conditional probability of the occurrence of the tth word calculated by a given language model based on the first t-1 words. The smaller the sentence perplexity ppl value, the more reasonable the sentence. Based on each word or phrase in the recommended candidate set, the corresponding recommended candidate sentence is obtained, and the sentence perplexity ppl is calculated for all the recommended candidate sentences, and they are arranged in order from small to large according to the ppl value. All the recommended candidate sentences that meet the following inequality are selected as the recommended text set:

[0130] ppl<ppl Top1 +C

[0131] Among them, C is a constant. The value of C can be adjusted according to the actual situation, or the sentences corresponding to the Top5 can be selected as the recommended text set. The selection method of the recommended text set can be selected according to the actual error correction scenario, and the recommended candidate sentence corresponding to the Top1 is used as the first candidate text.

[0132] Step S202: further calculate the recommended text set obtained in the previous step through the word granularity masked language model. The calculation method of the pseudo logarithmic maximum likelihood score of the word granularity masked language model is as follows:

[0133]

[0134] Among them, S represents the sentence, |S| represents the length of the sentence, and S ＜t =(w 1 , ..., w t-1 ), w i represents each word in the sentence, 1≤i≤|S| and i≠t, P LM (w t |S ＜t ) represents the conditional probability of the occurrence of the tth word calculated by the word-granularity masked language model based on the words other than the tth word, that is, the sum of the probabilities of the words in all masked positions is calculated based on the context of the entire sentence after masking. The recommended candidate sentence with the highest pseudo-logarithmic maximum likelihood score obtained by the word-granularity masked language model is used as the second candidate text.

[0135] Step S203: If the second candidate text is the text to be corrected, use it as the final recommended text to prevent excessive correction. If the recommended text set includes the text to be corrected, output the text to be corrected as the final recommended text, which is also to prevent excessive correction.

[0136] Step S204: If the text to be corrected is neither the second candidate text nor included in the recommended text set, determine whether the first candidate text and the second candidate text are the same. If they are the same, output the first candidate text as the final recommended text; if they are different, calculate the phonetic similarity between the first candidate text and the second candidate text and the text to be corrected. In this step, the Chinese phonetic similarity algorithm DimSim is used to calculate the phonetic similarity. If the obtained phonetic similarities are different, select the candidate text with the smaller phonetic similarity as the final recommended text. If the phonetic similarities are the same, select the candidate text with the least number of changes as the final recommended text. If the number of changes of the first candidate text and the second candidate text is also the same, output the first candidate text as the final recommended text.

[0137] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0138] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0139] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a text error correction device.

[0140] refer to Figure 3 , the text error correction device comprises:

[0141] The text acquisition module 301 is configured to acquire the text to be corrected;

[0142] The word segmentation module 302 is configured to segment the text to be corrected by a word segmentation algorithm to obtain a word segmentation correction text;

[0143] The error detection module 303 is configured to perform error detection on the text to be corrected and the word segmentation correction text respectively to obtain an error candidate set;

[0144] A replacement candidate word determination module 304 is configured to determine a replacement candidate set based on the error candidate set;

[0145] A grouping module 305 is configured to group the error candidate set to obtain multiple error candidate subsets;

[0146] A path determination module 306 is configured to generate a plurality of candidate paths based on a plurality of the error candidate subsets;

[0147] A recommendation set determination module 307, configured to determine a candidate recommendation set based on the replacement candidate set and the candidate paths;

[0148] The recommended text determination module 308 is configured to obtain the recommended text by using a preset algorithm based on the recommendation candidate set.

[0149] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0150] The device of the above embodiment is used to implement the corresponding text error correction method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0151] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the text error correction method described in any of the above embodiments is implemented.

[0152] Figure 4 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0153] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0154] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0155] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0156] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0157] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0158] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0159] The electronic device of the above embodiment is used to implement the corresponding text error correction method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0160] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the text correction method described in any of the above embodiments.

[0161] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0162] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the text error correction method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0163] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0164] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0165] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0166] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A text error correction method, characterized in that: include: Get the text to be corrected; Segment the text to be corrected by a word segmentation algorithm to obtain a word segmentation correction text; Performing error detection on the text to be corrected and the word segmentation correction text respectively to obtain an error candidate set, wherein the error candidate set includes error words; determining a replacement candidate set based on the error candidate set; Grouping the error candidate set to obtain multiple error candidate subsets; generating a plurality of candidate paths based on a plurality of said error candidate subsets; Determine a candidate recommendation set based on the replacement candidate set and the candidate paths; The recommended text is obtained based on the candidate recommendation set through a preset algorithm.

2. The error correction method according to claim 1, characterized in that: The word segmentation of the text to be corrected by using a word segmentation algorithm includes: The text to be corrected is segmented by Jieba word segmentation algorithm and CoreNLP word segmentation algorithm respectively.

3. The error correction method according to claim 1, characterized in that: The performing error detection on the text to be corrected and the word segmentation correction text respectively to obtain an error candidate set includes: Matching each character or word in the segmentation error correction text with a preset common word dictionary to obtain a first error set; Combining any two adjacent characters or words in the word segmentation error correction text in pairs to obtain a combination word set, searching the pinyin of each combination word in the combination word set in the common word dictionary to obtain a homophone set, and determining a second error set based on the homophone set; Outputting a predicted word set through a word granularity mask language model based on the text to be corrected, and determining a third error set based on the predicted word set; Matching each word in the text to be corrected with a preset confusion dictionary to obtain a fourth error set; The first error set, the second error set, the third error set, and the fourth error set are combined into the error candidate set.

4. The error correction method according to claim 3, characterized in that: The determining of a replacement candidate set based on the error candidate set comprises: Each character in each word in the error candidate set is replaced with all the characters in the common word dictionary in sequence to obtain a single-character replacement set, All characters in each word in the error candidate set are arranged and combined to obtain a rotation word set, and each character in each word in the rotation word set is replaced with all characters in the common word dictionary in sequence to obtain a rotation order set. Matching each word in the single-word replacement set and the rotation order set with the common word dictionary to obtain a first replacement set; Matching each character in the error candidate set with a preset dictionary of similar characters to obtain a second replacement set; Matching each character in the error candidate set with a preset dictionary of similar-sounding characters to obtain a third replacement set; The correct word group corresponding to each word in the fourth error set in the confusion dictionary is formed into a fourth replacement set; The first replacement set, the second replacement set, the third replacement set, the homonymous word set, the predicted word set and the fourth replacement set are combined into the replacement candidate set.

5. The error correction method according to claim 1, characterized in that: Before determining the candidate recommendation set based on the replacement candidate set and the candidate paths, the method further includes: The words or phrases in the replacement candidate set whose sentence perplexity corresponding to the words or phrases exceeds a preset threshold are removed from the replacement candidate set.

6. The error correction method according to claim 1, characterized in that: The generating a plurality of candidate paths based on the plurality of error candidate subsets comprises: All characters or words in the error candidate subsets are arranged and combined to obtain a plurality of candidate paths.

7. The error correction method according to claim 1, characterized in that: The determining of a candidate recommendation set based on the replacement candidate set and the candidate paths includes: The replacement candidate characters or words corresponding to each character or word in the candidate path in the replacement candidate set are combined into a candidate recommendation set corresponding to the candidate path.

8. The error correction method according to claim 1, characterized in that: The obtaining the recommended text based on the candidate recommendation set by using a preset algorithm includes: The recommended text is obtained by respectively calculating the candidate recommendation set using a sentence perplexity algorithm and a word granularity mask language model.

9. A text error correction device, characterized in that: include: A text acquisition module is configured to acquire the text to be corrected; A word segmentation module is configured to segment the text to be corrected by a word segmentation algorithm to obtain a word segmentation correction text; An error detection module is configured to perform error detection on the text to be corrected and the word segmentation correction text respectively to obtain an error candidate set; A replacement candidate word determination module, configured to determine a replacement candidate set based on the error candidate set; a grouping module, configured to group the error candidate set to obtain a plurality of error candidate subsets; A path determination module, configured to generate a plurality of candidate paths based on a plurality of the error candidate subsets; a recommendation set determination module, configured to determine a candidate recommendation set based on the replacement candidate set and the candidate paths; The recommended text determination module is configured to obtain the recommended text through a preset algorithm based on the candidate recommendation set.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images