A Seed-Based Method for Generating Typos Confusion Sets

Abstract

A method for generating a wrongly-written or mispronounced character confusion set based on seeds includes the following steps that (1) a wrongly-written or mispronounced character confusion set map is established, and the wrongly-written or mispronounced character confusion set map is established according to the seed wrongly-written or mispronounced character confusion set; (2) the wrongly-written or mispronounced character confusion set map is used, the law among wrongly-written or mispronounced characters can be automatically discovered and mined through an algorithm, and the wrongly-written or mispronounced characters are automatically added into the wrongly-written or mispronounced character confusion set; (3) homophonous wrongly-written or mispronounced characters are automatically generated in the wrongly-written or mispronounced character confusion set, and the homophonous wrongly-written or mispronounced Chinese characters are automatically added; (4) non-homophonous wrongly-written or mispronounced characters are automatically generated in the wrongly-written or mispronounced character confusion set, and the non-homophonous wrongly-written or mispronounced Chinese characters can be automatically added according to the features such as shape similarity and the wrongly-written or mispronounced character confusion set map.

Description

technical field [0001] The invention relates to natural language processing in the field of computers, in particular using a method based on a seed and a typo map to automatically establish a typo confusion set, effectively reducing the amount of labor, and the generated typo confusion set is effectively applied to an automatic proofreading system for Chinese texts. Background technique [0002] With the rapid development of information processing technology and the Internet, traditional text work is almost completely replaced by computers. Electronic texts such as e-books, e-newspapers, e-mails, and office documents, blogs, and microblogs have all become part of people's daily lives. However, there are more and more typos in the text, which brings great challenges to the proofreading work. Traditional manual proofreading has low efficiency, high intensity, and long cycle obviously cannot meet the needs of text proofreading. Therefore, it is necessary to study automatic tex...

AI Technical Summary

Problems solved by technology

[0003] Because the Chinese computer text is through the phonetic code input method (such as the Sogou Pinyin input method) and the shape code input method (such as the Wubi input method), so similar sound and shape are the main features of typos in Chinese characters. Generated by similarity algorithm or shape similarity algorithm, many very unreasonable confusion sets will be generated, which will lead to a very high false positive rate of the automatic proofreading system

If it is completely filtered manually, due to artificial subjectivity, some unreasonable confusion sets will be obtained and reasonable confusion sets will be missed, and the workload will be huge

Images