Entity-based lexical inspection method and device, computer equipment and storage medium

Abstract

The invention discloses an entity-based lexical examination method and device, computer equipment and a storage medium. The method comprises the steps of calculating word level N-Gram scores of threeadjacent words and word level N-Gram scores of three adjacent words; calculating an average absolute deviation between a word-level N-Gram score and a word-level N-Gram score, preliminarily identifying words with values greater than a threshold value as wrong words, and creating a wrong word set; counting the wrong words and creating a candidate set, respectively substituting the candidate words in the candidate set into the original text according to a sequence, and replacing the previous wrong words; performing word segmentation and character segmentation on the combined new text, calculating the sentence confusion degree of a word level by using a word level N-Gram, calculating the sentence confusion degree of a word level by using a word level N-Gram, and calculating the word average confusion degree of the sentence; and after calculating the confusion degree of the candidate words in the candidate set, comparing the confusion degree with the confusion degree of the original sentence, and selecting the candidate word with the minimum confusion degree as the optimal candidate word.

Description

technical field [0001] The invention relates to the technical field of statistical natural language processing, in particular to an entity-based lexical checking method and device, computer equipment and storage media. Background technique [0002] In the current technology, lexical detection is usually based on edit distance, language model, and dependency syntax to verify homonym typos, which is limited to the impact of the quality of language models and homonym thesaurus. The current grammar detection algorithm is limited to finding homonyms, and will There were a lot of misjudgments. Because the current grammar detection is mainly based on the recognition of a single word, it will also lead to the error correction of two words in a row. When using the edit distance to replace words, the error correction is often performed according to the word order, which leads to error correction. success rate is greatly reduced. [0003] The information disclosed in this Background ...

AI Technical Summary

Problems solved by technology

[0002] In the current technology, lexical detection is usually based on edit distance, language model, and dependency syntax to verify homonym typos, which is limited to the impact of the quality of language models and homonym thesaurus. The current grammar detection algorithm is limited to finding homonyms, and will A large number of misjudgments occurred

Because the current grammar detection is mainly based on the recognition of a single word, it will also lead to the error correction of two words in a row. When using the edit distance to replace words, the error correction is often performed according to the word order, which leads to error correction. The success rate is greatly reduced

Images