Method for automatically identifying word repetition errors

Abstract

The invention discloses a method for automatically identifying word repetition errors, which comprises the following steps of: after performing word segmentation on a large-scale training corpus, performing statistics to obtain a two-tuple structure and a three-tuple structure which comprise repeated words in the training corpus, and repeated combination degrees, left upper adjacent word information entropy and right lower adjacent word information entropy of the repeated words in the training corpus; counting and recording words containing repeated characters in the Chinese dictionary, and establishing a repeated character library of the Chinese dictionary; judging repeated words appearing in the text to be subjected to error checking based on the repeated words in the Chinese dictionary;and judging repeated words appearing in the to-be-debugged text based on the repeated combination degree, the left upper text adjacent word information entropy and the right lower text adjacent wordinformation entropy which are obtained through statistics. According to the method, whether the repeated words are the repeated words recorded in the dictionary or not can be quickly judged and recognized, whether the repeated words are not the repeated words in the dictionary but belong to the repeated words in daily terms or not can be effectively judged, judgment and recognition are quick and comprehensive, and practicability is high.

Description

technical field [0001] The invention relates to a natural language processing method, in particular to a method for discovering word repetition errors in the field of Chinese automatic proofreading. Background technique [0002] In the era of big data, there are more and more text data, and there are more and more errors in the text, including word repetition errors (also known as insertion errors). In Chinese, some words can be repeated, such as "research research", but some words cannot be repeated, such as "apology, apology" and "de", once they appear, it is a repetition error. [0003] How to automatically find the repetition of words in the text is one of the research contents of automatic proofreading of Chinese text. [0004] However, there is such a phenomenon of reasonable repetition of words in Chinese, so simple judgment of repeated words will bring a lot of misjudgments, and now most of the automatic proofreading of Chinese texts does not deal with word repetiti...

AI Technical Summary

Problems solved by technology

[0004] However, there is such a phenomenon of reasonable repetition of words in Chinese, so simple judgment of repeated words will bring a lot of misjudgments, and now most of the automatic proofreading of Chinese texts does not deal with word repetition errors separately, but simply Use binary or ternary information of words to judge whether there is an error

Images