Multilingual word segmentation method based on dictionaries and grammar analysis

Abstract

The invention discloses a multilingual word segmentation method based on dictionaries and grammar analysis. Efficient and accurate word segmentation of mixed texts of Chinese, Japanese, Korean, Cantonese and the like can be realized, flexible lexicon expansion of words for different time periods and different professionals can be realized, lexicon information is updated effectively, and efficient and accurate multilingual language text word segmentation is realized; a word segmentation sub-device of Chinese, Japanese, Korean, Cantonese and other language families, a Chinese quantum word segmentation device and a western language word segmentation device are embedded to realize the accurate word segmentation of each language text; a text segment to be performed with word segmentation is segmented by a built-in language segment coded identification mechanism, each segmented text segment corresponds to a language family, and the word segmentation is carried out by using a corresponding word segmentation sub-device; the word segmentation of western inflectional languages and the smart mode word segmentation of the Chinese, Japanese, Korean, Cantonese can be realized by grammar analysis, and texts containing Arabic numeral information can be processed; and meanwhile, the word segmentation of texts with a plurality of mixed languages can also be realized by the multilingual word segmentation method provided by the invention, thereby getting rid of the limitation that a word segmentation tool can only realize the word segmentation of single language and some individual languages and ensuring the security, accuracy, efficiency, flexibility and universality of word segmentation of texts. The multilingual word segmentation method provided by the invention has a wide application prospect in the text word segmentation fields such as enhancement of mass data text classification, text information extraction, autoabstract, etc.

Description

technical field [0001] The invention belongs to the field of natural language processing, and in particular relates to a multilingual word segmentation method based on dictionary and grammar analysis which is discriminated by unicode (unicode, universal code, single code) encoding. Background technique [0002] With the advent of the information age, more and more information can be viewed and retrieved by people, and the search market value continues to increase, more and more enterprises are looking for a more powerful natural language processing tool, such as automatic summarization, automatic Text retrieval, automatic text classification and other language processing tools, and automatic word segmentation technology is one of the core technologies of these tools. Word segmentation, as the name implies, is to automatically segment the text with the help of a computer, so that it can correctly express the meaning to be expressed without losing information. As long as it i...

AI Technical Summary

Problems solved by technology

[0004] The present invention provides a multilingual word segmentation method and system based on dictionaries and grammatical analysis, which overcomes the limitation that only a single language or individual languages ​​can be segmented, and uses word segmentation technology based on dictionary matching and grammatical analysis to realize different languages. The purpose of word segmentation is to ensure that the text can be efficiently decomposed into representative words. For some users, there is a need to accurately decompose the text content, that is, some ambiguous words can be disambiguated. Therefore, the present invention adopts the method of grammatical analysis to disambiguate and analyze the ambiguous words matched by the dictionary. In addition, some garbled characters or some stop words with little representative meaning may appear in the text to be segmented. The present invention will filter it to ensure the readability and efficient searchability of the text, and at the same time reduce the storage space required for the text

Images