Text error correction method and system

Abstract

The invention discloses a text error correction method and system. The method comprises the steps of training a BERT model; encoding the Chinese characters; inputting a sentence to be detected into the BERT model to obtain a confidence coefficient TopK candidate set at each position; calculating the similarity between the candidate item and the original item by using the Chinese character code; calculating an error correction probability by utilizing the similarity and the confidence coefficient; reordering the candidate set according to the error correction probability; comparing the probability with a set threshold value, and if the probability is lower than the threshold value, not performing error correction; and if not, taking out the Top1 in the candidate set as a final error correction result. According to the method, the problem that a large number of training corpora are lacked in an existing technical scheme is solved by using BERT, the influence of the spelling fonts of theChinese characters on the error text is comprehensively considered by using a novel Chinese character encoding mode, and the method is dynamically suitable for various error correction tasks.

Description

technical field [0001] The invention relates to the field of language processing, in particular to a text error correction method and system. Background technique [0002] Existing text error correction methods are mainly divided into statistical models and neural network models. The method based on the statistical model takes N-gram as an example, by calculating the adjacent N-gram probability with the target word as the core in the sentence, the error detection and judgment of the target word are performed, and the confusion set is sorted at the same time, and the first candidate word is selected for correction. error text. The method based on the neural network model takes the Seq2Seq model as an example. By inputting the sentence into the Encoder, after the model is calculated, the Decoder will output the character with a confidence score of Top1 at each position in the sentence for error correction. [0003] The above prior art has the following disadvantages: (1) The...

AI Technical Summary

Problems solved by technology

[0004] The present invention aims at the shortcomings in the prior art, solves the problem of lacking a large amount of training corpus in the prior art solution by using BERT, and utilizes a new Chinese character encoding method, comprehensively considers the impact of Chinese spelling fonts on error text, and dynamically adapts to various Error correction task

Images